[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:15 dist-test-slave-wl2h krb5kdc[28932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:15 dist-test-slave-wl2h krb5kdc[28932](info): set up 2 sockets
Aug 09 16:39:15 dist-test-slave-wl2h krb5kdc[28932](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 16:39:17.033299 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:17.175590 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:41803
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:41803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:17.272833 28946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:17.273155 28946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:17.273208 28946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:17.274606 28946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:17.274677 28946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:17.274703 28946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:17.274726 28946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:17.274757 28946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:17.276640 28946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:41803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:17.276957 28946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:17.277240 28946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:17.283947 28955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:17.283955 28957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.284003 28946 server_base.cc:1034] running on GCE node
W20260809 16:39:17.283985 28952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.284572 28946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:17.284869 28946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:17.286018 28946 hybrid_clock.cc:648] HybridClock initialized: now 1786293557286002 us; error 38 us; skew 500 ppm
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:17.289117 28946 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:17.289531 28946 webserver.cc:460] Webserver started at http://127.28.62.190:45511/ using document root <none> and password file <none>
I20260809 16:39:17.289752 28946 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:17.289798 28946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:17.289892 28946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:17.291070 28946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "6f229635f7fe45aab484c0e59bc56456"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "e133279b9bb3137a90fd1c43d63df498"
server_key_iv: "4fff3a6163e0c74cf297edb16436652b"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.291476 28946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "6f229635f7fe45aab484c0e59bc56456"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "e133279b9bb3137a90fd1c43d63df498"
server_key_iv: "4fff3a6163e0c74cf297edb16436652b"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.292981 28946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:39:17.293776 28965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.293969 28946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:17.294044 28946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "6f229635f7fe45aab484c0e59bc56456"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "e133279b9bb3137a90fd1c43d63df498"
server_key_iv: "4fff3a6163e0c74cf297edb16436652b"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.294097 28946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:17.302927 28946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:17.305405 28946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:17.305522 28946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:17.314551 28946 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:41803
I20260809 16:39:17.314729 29023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:41803 every 8 connection(s)
I20260809 16:39:17.315114 28946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:17.316347 29024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:17.319741 29024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456: Bootstrap starting.
I20260809 16:39:17.320923 29024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:17.321648 29024 log.cc:824] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:17.322645 29024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456: No bootstrap required, opened a new log
I20260809 16:39:17.322675 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 28946
I20260809 16:39:17.322963 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:17.323751 28922 external_mini_cluster.cc:1428] Setting key cb190db1b1993950bad73669fc17deb2
I20260809 16:39:17.326268 29024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } }
I20260809 16:39:17.326447 29024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:17.326498 29024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f229635f7fe45aab484c0e59bc56456, State: Initialized, Role: FOLLOWER
I20260809 16:39:17.326658 29024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } }
I20260809 16:39:17.326754 29024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:17.326799 29024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:17.326849 29024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:17.328104 29024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } }
I20260809 16:39:17.328239 29024 leader_election.cc:302] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f229635f7fe45aab484c0e59bc56456; no voters:
I20260809 16:39:17.328433 29024 leader_election.cc:288] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:17.328521 29030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:17.328706 29024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:17.328822 29030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [term 1 LEADER]: Becoming Leader. State: Replica: 6f229635f7fe45aab484c0e59bc56456, State: Running, Role: LEADER
I20260809 16:39:17.328928 29030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } }
I20260809 16:39:17.329516 29032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f229635f7fe45aab484c0e59bc56456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } } }
I20260809 16:39:17.329674 29032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:17.329504 29035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f229635f7fe45aab484c0e59bc56456. Latest consensus state: current_term: 1 leader_uuid: "6f229635f7fe45aab484c0e59bc56456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f229635f7fe45aab484c0e59bc56456" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 41803 } } }
I20260809 16:39:17.329764 29035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:17.330317 29043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:17.331759 29043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:17.334419 29043 catalog_manager.cc:1347] Generated new cluster ID: 7bcb41b9614c4e34904715c1daa86c11
I20260809 16:39:17.334475 29043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:17.341077 29043 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:17.341701 29043 catalog_manager.cc:1504] Loading token signing keys...
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:17.348315 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.324830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.28.62.190:41803)
0809 16:39:17.325171 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.325182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.325454 (+ 272us) server_negotiation.cc:407] Connection header received
0809 16:39:17.328750 (+ 3296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.328767 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.328833 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.328971 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.332163 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.334933 (+ 2770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.336871 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.337603 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.337722 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.343251 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.343256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.343256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.343269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.345982 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.346498 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.346501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.346501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.346656 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.346952 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.346952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.346953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.347334 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.347655 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.347752 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.347955 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":145,"threads_started":1}
I20260809 16:39:17.350674 29043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f229635f7fe45aab484c0e59bc56456: Generated new TSK 0
I20260809 16:39:17.350865 29043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:17.386860 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:17.471556 29058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:17.471778 29058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:17.471804 29058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:17.473083 29058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:17.473129 29058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:17.473155 29058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:17.474583 29058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:17.474815 29058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:17.475113 29058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:17.481034 29064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.481178 29058 server_base.cc:1034] running on GCE node
W20260809 16:39:17.481045 29063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:17.481060 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.481478 29058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:17.481721 29058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:17.482892 29058 hybrid_clock.cc:648] HybridClock initialized: now 1786293557482863 us; error 38 us; skew 500 ppm
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:17.485440 29058 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:17.485867 29058 webserver.cc:460] Webserver started at http://127.28.62.129:42539/ using document root <none> and password file <none>
I20260809 16:39:17.486094 29058 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:17.486164 29058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:17.486307 29058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:17.487429 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "59494dbcd66d40bf99522155a2832f58"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "a1bd56b7433c0754ea200f276fc72050"
server_key_iv: "e818e69dd5bd0543e5493c5ad0c28019"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.487818 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "59494dbcd66d40bf99522155a2832f58"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "a1bd56b7433c0754ea200f276fc72050"
server_key_iv: "e818e69dd5bd0543e5493c5ad0c28019"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.489148 29058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:17.490082 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.490298 29058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:17.490396 29058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "59494dbcd66d40bf99522155a2832f58"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "a1bd56b7433c0754ea200f276fc72050"
server_key_iv: "e818e69dd5bd0543e5493c5ad0c28019"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.490475 29058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:17.510452 29058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:17.512549 29058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:17.512727 29058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:17.512989 29058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:17.513316 29058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:17.513370 29058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.513406 29058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:17.513445 29058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.520264 29058 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:34115
I20260809 16:39:17.520315 29211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:34115 every 8 connection(s)
I20260809 16:39:17.520709 29058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:17.522487 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29058
I20260809 16:39:17.522560 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:17.522719 28922 external_mini_cluster.cc:1428] Setting key 8b977c9d69162d7ec00a250d45ed0a7a
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:17.531366 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.521436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:37001 (local address 127.28.62.190:41803)
0809 16:39:17.521503 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.521505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.521709 (+ 204us) server_negotiation.cc:407] Connection header received
0809 16:39:17.522188 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.522189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.522199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.522281 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.523814 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.524316 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.525618 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.525819 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.525871 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.528368 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.528369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.528370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.528378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.529899 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.530302 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.530303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.530303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.530366 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.530651 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.530652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.530652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.530779 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.530859 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.531224 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.531300 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 16:39:17.531677 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.521417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:41803 (local address 127.28.62.129:37001)
0809 16:39:17.521591 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:17.521627 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:17.522110 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:17.522358 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:17.522361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:17.523492 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:17.523701 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.523705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.524412 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.524413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.525515 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.525518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.525904 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.525905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.526051 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.526582 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:17.526590 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:17.528277 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:17.529975 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.529983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.529986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.530209 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.530427 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.530428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.530428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.530574 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.530870 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:17.530879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:17.531079 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 16:39:17.531213 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 16:39:17.531395 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:39:17.532298 29212 heartbeater.cc:347] Connected to a master server at 127.28.62.190:41803
I20260809 16:39:17.532397 29212 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:17.532610 29212 heartbeater.cc:511] Master 127.28.62.190:41803 requested a full tablet report, sending...
I20260809 16:39:17.533093 28984 ts_manager.cc:194] Registered new tserver with Master: 59494dbcd66d40bf99522155a2832f58 (127.28.62.129:34115)
I20260809 16:39:17.533762 28984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:37001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:17.551898 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:17.636595 29218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:17.636826 29218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:17.636862 29218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:17.637985 29218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:17.638051 29218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:17.638103 29218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:17.639616 29218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:17.639891 29218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:17.640125 29218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:17.645210 29226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.645358 29218 server_base.cc:1034] running on GCE node
W20260809 16:39:17.645421 29225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:17.645483 29228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.645680 29218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:17.645882 29218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:17.647055 29218 hybrid_clock.cc:648] HybridClock initialized: now 1786293557647041 us; error 28 us; skew 500 ppm
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:17.649637 29218 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:17.650058 29218 webserver.cc:460] Webserver started at http://127.28.62.130:33985/ using document root <none> and password file <none>
I20260809 16:39:17.650337 29218 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:17.650384 29218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:17.650543 29218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:17.651640 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "2dd0727e8a9a43c0a6a79438552f40ad"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "115852d1de4052ee8a7798db7d112af9"
server_key_iv: "a689a8adc1eaab66067f78bae8990471"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.651997 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "2dd0727e8a9a43c0a6a79438552f40ad"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "115852d1de4052ee8a7798db7d112af9"
server_key_iv: "a689a8adc1eaab66067f78bae8990471"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.653441 29218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:17.654268 29236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.654493 29218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:17.654567 29218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "2dd0727e8a9a43c0a6a79438552f40ad"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "115852d1de4052ee8a7798db7d112af9"
server_key_iv: "a689a8adc1eaab66067f78bae8990471"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.654659 29218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:17.679721 29218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:17.681685 29218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:17.681841 29218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:17.682085 29218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:17.682407 29218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:17.682440 29218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.682482 29218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:17.682509 29218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.688580 29218 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:38223
I20260809 16:39:17.688657 29366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:38223 every 8 connection(s)
I20260809 16:39:17.688977 29218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:17.697634 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29218
I20260809 16:39:17.697736 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:17.697908 28922 external_mini_cluster.cc:1428] Setting key 3b7278fbf46a78c4a05db2f1573b00d3
I20260809 16:39:17.698559 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.689725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:36597 (local address 127.28.62.190:41803)
0809 16:39:17.689775 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.689777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.689980 (+ 203us) server_negotiation.cc:407] Connection header received
0809 16:39:17.690473 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.690474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.690485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.690542 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.691539 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.692035 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.693006 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.693176 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.693273 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.695519 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.695521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.695521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.695528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.696921 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.697336 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.697337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.697337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.697390 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.697704 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.697705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.697705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.697881 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.697957 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.698363 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.698432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:39:17.698732 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.689680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:41803 (local address 127.28.62.130:36597)
0809 16:39:17.689854 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:17.689897 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:17.690392 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:17.690595 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:17.690598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:17.691263 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:17.691462 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.691467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.692122 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.692123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.692935 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.692937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.693286 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.693287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.693420 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.693936 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:17.693944 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:17.695432 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:17.696991 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.696996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.696998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.697262 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.697503 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.697504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.697505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.697632 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.697959 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:17.697966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:17.698209 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 16:39:17.698330 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 16:39:17.698493 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260809 16:39:17.699195 29367 heartbeater.cc:347] Connected to a master server at 127.28.62.190:41803
I20260809 16:39:17.699286 29367 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:17.699492 29367 heartbeater.cc:511] Master 127.28.62.190:41803 requested a full tablet report, sending...
I20260809 16:39:17.699889 28984 ts_manager.cc:194] Registered new tserver with Master: 2dd0727e8a9a43c0a6a79438552f40ad (127.28.62.130:38223)
I20260809 16:39:17.700266 28984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:36597
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:17.725324 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:17.808175 29376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:17.808421 29376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:17.808463 29376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:17.809612 29376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:17.809676 29376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:17.809722 29376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:17.811510 29376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:41803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:17.811784 29376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:17.812073 29376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:17.817351 29386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:17.817363 29383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:17.817487 29384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:17.817564 29376 server_base.cc:1034] running on GCE node
I20260809 16:39:17.817844 29376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:17.818097 29376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:17.819259 29376 hybrid_clock.cc:648] HybridClock initialized: now 1786293557819254 us; error 35 us; skew 500 ppm
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:17.821926 29376 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:17.822481 29376 webserver.cc:460] Webserver started at http://127.28.62.131:33795/ using document root <none> and password file <none>
I20260809 16:39:17.822713 29376 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:17.822779 29376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:17.822932 29376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:17.824025 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "9531bc5a6a1c447c8c3c62c8742d0194"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "4db8b4baba270d5c8f1808f3299cf743"
server_key_iv: "135ac7e637ec122650446cdbd71abbe5"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.824477 29376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "9531bc5a6a1c447c8c3c62c8742d0194"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "4db8b4baba270d5c8f1808f3299cf743"
server_key_iv: "135ac7e637ec122650446cdbd71abbe5"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.825877 29376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:17.826746 29393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.826946 29376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:17.827066 29376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "9531bc5a6a1c447c8c3c62c8742d0194"
format_stamp: "Formatted at 2026-08-09 16:39:17 on dist-test-slave-wl2h"
server_key: "4db8b4baba270d5c8f1808f3299cf743"
server_key_iv: "135ac7e637ec122650446cdbd71abbe5"
server_key_version: "encryptionkey@0"
I20260809 16:39:17.827190 29376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:17.835531 29376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:17.837574 29376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:17.837754 29376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:17.837994 29376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:17.838322 29376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:17.838376 29376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.838418 29376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:17.838443 29376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:17.845027 29376 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:43459
I20260809 16:39:17.845072 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:43459 every 8 connection(s)
I20260809 16:39:17.845398 29376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:17.850317 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29376
I20260809 16:39:17.850399 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:17.850608 28922 external_mini_cluster.cc:1428] Setting key 67929e90900d2776a53222d903b6dd69
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:17.855245 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.846164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:36395 (local address 127.28.62.190:41803)
0809 16:39:17.846226 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.846229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.846463 (+ 234us) server_negotiation.cc:407] Connection header received
0809 16:39:17.846989 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.846991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.847004 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.847089 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.848041 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.848574 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.849580 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.849790 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.849864 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.852655 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.852657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.852657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.852665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.853857 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.854259 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.854261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.854261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.854321 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.854580 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.854581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.854581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.854704 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.854779 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.855070 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.855155 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 16:39:17.855438 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.846116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:41803 (local address 127.28.62.131:36395)
0809 16:39:17.846323 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:17.846373 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:17.846890 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:17.847162 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:17.847167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:17.847744 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:17.847944 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.847948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.848678 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.848679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.849480 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:17.849482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.849872 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.849872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:17.849994 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.850591 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:17.850601 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:17.852540 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:17.853949 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.853955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.853958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.854183 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.854394 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:17.854395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:17.854395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:17.854502 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:17.854768 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:17.854774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:17.854913 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 16:39:17.855013 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 16:39:17.855199 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260809 16:39:17.855939 29520 heartbeater.cc:347] Connected to a master server at 127.28.62.190:41803
I20260809 16:39:17.856040 29520 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:17.856252 29520 heartbeater.cc:511] Master 127.28.62.190:41803 requested a full tablet report, sending...
I20260809 16:39:17.856617 28984 ts_manager.cc:194] Registered new tserver with Master: 9531bc5a6a1c447c8c3c62c8742d0194 (127.28.62.131:43459)
I20260809 16:39:17.856952 28984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:36395
I20260809 16:39:17.863090 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:17.887739 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.879963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.28.62.129:34115)
0809 16:39:17.880254 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.880259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.880271 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:17.880298 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.880299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.880314 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.880386 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.880939 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.881674 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.882377 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.882695 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.882750 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.885012 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.885019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.885019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.885026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.886617 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.886939 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.886941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.886942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.886995 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.887283 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.887284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.887284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.887417 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.887521 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.887584 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.887658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
W20260809 16:39:17.888172 29111 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46080
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:17.912590 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:17.904979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.28.62.129:34115)
0809 16:39:17.905050 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:17.905052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:17.905113 (+ 61us) server_negotiation.cc:407] Connection header received
0809 16:39:17.905172 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:17.905173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:17.905186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:17.905238 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:17.905800 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.906310 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.907081 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:17.907304 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:17.907364 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:17.909942 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:17.909944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:17.909944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:17.909951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:17.911479 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.911796 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.911797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.911797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.911846 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:17.912082 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:17.912083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:17.912084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:17.912204 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:17.912312 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:17.912419 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 16:39:17.912486 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 16:39:17.913291 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29058
I20260809 16:39:17.925624 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29218
I20260809 16:39:17.933171 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29376
I20260809 16:39:17.939646 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 28946
2026-08-09T16:39:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2154 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[29538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:17 dist-test-slave-wl2h krb5kdc[29538](info): set up 2 sockets
Aug 09 16:39:18 dist-test-slave-wl2h krb5kdc[29538](info): commencing operation
krb5kdc: starting...
W20260809 16:39:19.113036 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:19.241722 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46227
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:19.325310 29553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:19.325533 29553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:19.325557 29553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:19.326746 29553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:19.326794 29553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:19.326809 29553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:19.326822 29553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:19.326876 29553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:19.328438 29553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:19.328713 29553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:19.328958 29553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:39:19.334304 29553 server_base.cc:1034] running on GCE node
W20260809 16:39:19.334263 29559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.334271 29563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.334276 29560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:19.334756 29553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:19.335135 29553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:19.336303 29553 hybrid_clock.cc:648] HybridClock initialized: now 1786293559336311 us; error 48 us; skew 500 ppm
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:19.338773 29553 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:19.339277 29553 webserver.cc:460] Webserver started at http://127.28.62.190:45635/ using document root <none> and password file <none>
I20260809 16:39:19.339529 29553 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:19.339594 29553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:19.339753 29553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:19.340816 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "2830ab3b150e4c5f925670b8539e7efb"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "a4d541ac5cd9fa0c2d3267c64e801b75"
server_key_iv: "40bebbbf306dd6c16cd230d930f7df44"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.341193 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "2830ab3b150e4c5f925670b8539e7efb"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "a4d541ac5cd9fa0c2d3267c64e801b75"
server_key_iv: "40bebbbf306dd6c16cd230d930f7df44"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.342525 29553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:19.343271 29571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.343438 29553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:19.343542 29553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "2830ab3b150e4c5f925670b8539e7efb"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "a4d541ac5cd9fa0c2d3267c64e801b75"
server_key_iv: "40bebbbf306dd6c16cd230d930f7df44"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.343616 29553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:19.350075 29553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:19.351943 29553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:19.352119 29553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:19.356426 29553 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:46227
I20260809 16:39:19.356462 29631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:46227 every 8 connection(s)
I20260809 16:39:19.356834 29553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:19.357668 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:19.359990 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb: Bootstrap starting.
I20260809 16:39:19.360514 29632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:19.360775 29632 log.cc:824] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:19.361439 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb: No bootstrap required, opened a new log
I20260809 16:39:19.363003 29632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } }
I20260809 16:39:19.363159 29632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:19.363183 29632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2830ab3b150e4c5f925670b8539e7efb, State: Initialized, Role: FOLLOWER
I20260809 16:39:19.363283 29632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } }
I20260809 16:39:19.363335 29632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:19.363356 29632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:19.363432 29632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:19.363966 29632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } }
I20260809 16:39:19.364079 29632 leader_election.cc:302] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2830ab3b150e4c5f925670b8539e7efb; no voters:
I20260809 16:39:19.364257 29632 leader_election.cc:288] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:19.364313 29636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:19.364522 29632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:19.364526 29636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [term 1 LEADER]: Becoming Leader. State: Replica: 2830ab3b150e4c5f925670b8539e7efb, State: Running, Role: LEADER
I20260809 16:39:19.364662 29636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } }
I20260809 16:39:19.365244 29638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2830ab3b150e4c5f925670b8539e7efb. Latest consensus state: current_term: 1 leader_uuid: "2830ab3b150e4c5f925670b8539e7efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } } }
I20260809 16:39:19.365352 29638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:19.365518 29637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2830ab3b150e4c5f925670b8539e7efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2830ab3b150e4c5f925670b8539e7efb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46227 } } }
I20260809 16:39:19.365597 29637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:19.365655 29646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:19.366216 29646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:19.367053 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29553
I20260809 16:39:19.367142 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:19.367318 28922 external_mini_cluster.cc:1428] Setting key 8eff6b8676f3d02607184dec64aa315f
I20260809 16:39:19.368271 29646 catalog_manager.cc:1347] Generated new cluster ID: 84da7aef756d46f496be9bc2e7277a84
I20260809 16:39:19.368323 29646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:19.379441 29646 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:19.380223 29646 catalog_manager.cc:1504] Loading token signing keys...
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:19.387351 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.367816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.28.62.190:46227)
0809 16:39:19.371162 (+ 3346us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.371172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.371223 (+ 51us) server_negotiation.cc:407] Connection header received
0809 16:39:19.371350 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.371358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.371417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.371565 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:19.372314 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.374168 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.375221 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.375896 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.376007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.383576 (+ 7569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:19.383581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:19.383581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:19.383594 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:19.385602 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.386050 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.386054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.386054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.386159 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.386373 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.386374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.386374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.386668 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:19.386852 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.386949 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.387131 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3279,"thread_start_us":95,"threads_started":1}
I20260809 16:39:19.387594 29646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2830ab3b150e4c5f925670b8539e7efb: Generated new TSK 0
I20260809 16:39:19.387777 29646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:19.425629 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:19.523816 29661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:19.524029 29661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:19.524055 29661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:19.525192 29661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:19.525230 29661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:19.525257 29661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:19.526681 29661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:19.526881 29661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:19.527154 29661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:19.532256 29671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.532322 29666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.532258 29668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:19.532320 29661 server_base.cc:1034] running on GCE node
I20260809 16:39:19.532678 29661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:19.532895 29661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:19.534040 29661 hybrid_clock.cc:648] HybridClock initialized: now 1786293559534025 us; error 28 us; skew 500 ppm
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:19.536597 29661 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:19.536983 29661 webserver.cc:460] Webserver started at http://127.28.62.129:37719/ using document root <none> and password file <none>
I20260809 16:39:19.537164 29661 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:19.537211 29661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:19.537299 29661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:19.538287 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "0670ee246ee8480e83f42ffc6739fe16"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "437fb7329a925020ddadde7c335a4a83"
server_key_iv: "61597a2f5b608691815878ab5f5bf439"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.538581 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "0670ee246ee8480e83f42ffc6739fe16"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "437fb7329a925020ddadde7c335a4a83"
server_key_iv: "61597a2f5b608691815878ab5f5bf439"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.539954 29661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:19.540750 29679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.540936 29661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:19.541013 29661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "0670ee246ee8480e83f42ffc6739fe16"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "437fb7329a925020ddadde7c335a4a83"
server_key_iv: "61597a2f5b608691815878ab5f5bf439"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.541066 29661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:19.553562 29661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:19.555668 29661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:19.555790 29661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:19.555999 29661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:19.556267 29661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:19.556298 29661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.556321 29661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:19.556334 29661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.562383 29661 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:38237
I20260809 16:39:19.562462 29815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:38237 every 8 connection(s)
I20260809 16:39:19.562778 29661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:19.572288 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29661
I20260809 16:39:19.572397 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:19.572573 28922 external_mini_cluster.cc:1428] Setting key 69559d18b0b87a0af787f456197060a9
I20260809 16:39:19.572642 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.563591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:40511 (local address 127.28.62.190:46227)
0809 16:39:19.563673 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.563676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.563859 (+ 183us) server_negotiation.cc:407] Connection header received
0809 16:39:19.564940 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.564941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.564952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.565008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:19.565693 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.566284 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.567282 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.567489 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.567546 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.569777 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:19.569778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:19.569779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:19.569786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:19.571159 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.571533 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.571534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.571534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.571593 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.571844 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.571845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.571845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.571973 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:19.572067 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.572444 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.572550 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 16:39:19.572803 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.563558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46227 (local address 127.28.62.129:40511)
0809 16:39:19.563741 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.563778 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.564845 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.565060 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.565063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.565370 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:19.565587 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.565593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.566363 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.566364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.567202 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.567205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.567562 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.567562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.567661 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.568222 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:19.568231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:19.569665 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:19.571231 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.571238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.571240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.571466 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.571658 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.571659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.571659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.571759 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.572037 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:19.572062 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:19.572236 (+ 174us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.572367 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.572554 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260809 16:39:19.573331 29817 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46227
I20260809 16:39:19.573451 29817 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:19.573673 29817 heartbeater.cc:511] Master 127.28.62.190:46227 requested a full tablet report, sending...
I20260809 16:39:19.574258 29592 ts_manager.cc:194] Registered new tserver with Master: 0670ee246ee8480e83f42ffc6739fe16 (127.28.62.129:38237)
I20260809 16:39:19.575235 29592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:40511
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:19.601682 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:19.685774 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:19.686028 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:19.686067 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:19.687217 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:19.687292 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:19.687340 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:19.688805 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:19.689070 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:19.689311 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:19.694442 29830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.694432 29832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.694463 29834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:19.694777 29823 server_base.cc:1034] running on GCE node
I20260809 16:39:19.694976 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:19.695242 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:19.696406 29823 hybrid_clock.cc:648] HybridClock initialized: now 1786293559696399 us; error 37 us; skew 500 ppm
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:19.698937 29823 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:19.699448 29823 webserver.cc:460] Webserver started at http://127.28.62.130:43743/ using document root <none> and password file <none>
I20260809 16:39:19.699678 29823 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:19.699743 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:19.699891 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:19.700974 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "7b2de714cedd4162829e5982578661ff"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "30ffe1a808b787a3dd9580281958e85c"
server_key_iv: "8b89f862ca152c3671554b21ee63f308"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.701351 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "7b2de714cedd4162829e5982578661ff"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "30ffe1a808b787a3dd9580281958e85c"
server_key_iv: "8b89f862ca152c3671554b21ee63f308"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.702793 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:19.703665 29845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.703858 29823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:19.703958 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "7b2de714cedd4162829e5982578661ff"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "30ffe1a808b787a3dd9580281958e85c"
server_key_iv: "8b89f862ca152c3671554b21ee63f308"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.704042 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:19.712324 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:19.714180 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:19.714361 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:19.714603 29823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:19.714920 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:19.714973 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.715014 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:19.715128 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.722201 29823 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:45059
I20260809 16:39:19.722259 29977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:45059 every 8 connection(s)
I20260809 16:39:19.722581 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:19.727226 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29823
I20260809 16:39:19.727306 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:19.727471 28922 external_mini_cluster.cc:1428] Setting key 1ad5cb82229dad89f7bfaa023372c276
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:19.732649 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.723365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60531 (local address 127.28.62.190:46227)
0809 16:39:19.723428 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.723430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.723612 (+ 182us) server_negotiation.cc:407] Connection header received
0809 16:39:19.724540 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.724541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.724553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.724614 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:19.725189 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.725679 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.726762 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.726955 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.727067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.729577 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:19.729579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:19.729579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:19.729587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:19.731124 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.731627 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.731628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.731629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.731680 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.731950 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.731951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.731951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.732063 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:19.732115 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.732462 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.732536 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:19.732932 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.723325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46227 (local address 127.28.62.130:60531)
0809 16:39:19.723489 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.723532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.724440 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.724674 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.724677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.724927 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:19.725119 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.725122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.725793 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.725794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.726676 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.726678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.727083 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.727085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.727234 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.727871 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:19.727880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:19.729488 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:19.731201 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.731208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.731211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.731538 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.731741 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.731742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.731742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.731887 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.732125 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:19.732131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:19.732304 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.732436 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.732637 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260809 16:39:19.733465 29978 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46227
I20260809 16:39:19.733556 29978 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:19.733748 29978 heartbeater.cc:511] Master 127.28.62.190:46227 requested a full tablet report, sending...
I20260809 16:39:19.734126 29592 ts_manager.cc:194] Registered new tserver with Master: 7b2de714cedd4162829e5982578661ff (127.28.62.130:45059)
I20260809 16:39:19.734477 29592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:60531
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:19.757961 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:19.841979 29984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:19.842227 29984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:19.842269 29984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:19.843473 29984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:19.843536 29984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:19.843585 29984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:19.845168 29984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:19.845423 29984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:19.845650 29984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:19.850894 29994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.850886 29992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:19.850881 29990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:19.851215 29984 server_base.cc:1034] running on GCE node
I20260809 16:39:19.851459 29984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:19.851701 29984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:19.852869 29984 hybrid_clock.cc:648] HybridClock initialized: now 1786293559852862 us; error 41 us; skew 500 ppm
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:19.855526 29984 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:19.856037 29984 webserver.cc:460] Webserver started at http://127.28.62.131:43297/ using document root <none> and password file <none>
I20260809 16:39:19.856277 29984 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:19.856344 29984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:19.856487 29984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:19.857558 29984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "7a7db8e6b83c430185e6e71c174e4a22"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "d8add5b217b9e5d2e01ebef11405d161"
server_key_iv: "06d1485003eaf95ab86780774c0364ae"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.857937 29984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7db8e6b83c430185e6e71c174e4a22"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "d8add5b217b9e5d2e01ebef11405d161"
server_key_iv: "06d1485003eaf95ab86780774c0364ae"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.859330 29984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:39:19.860121 30002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.860306 29984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:19.860410 29984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "7a7db8e6b83c430185e6e71c174e4a22"
format_stamp: "Formatted at 2026-08-09 16:39:19 on dist-test-slave-wl2h"
server_key: "d8add5b217b9e5d2e01ebef11405d161"
server_key_iv: "06d1485003eaf95ab86780774c0364ae"
server_key_version: "encryptionkey@0"
I20260809 16:39:19.860483 29984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:19.874790 29984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:19.877233 29984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:19.877411 29984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:19.877665 29984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:19.877995 29984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:19.878048 29984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.878098 29984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:19.878125 29984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:19.884383 29984 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:33655
I20260809 16:39:19.884630 30124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:33655 every 8 connection(s)
I20260809 16:39:19.884900 29984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:39:19 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:19.893857 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 29984
I20260809 16:39:19.893963 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:19.894134 28922 external_mini_cluster.cc:1428] Setting key f287ff983d93cff8ca3494db3e2ffb4b
I20260809 16:39:19.895453 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.885784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:32853 (local address 127.28.62.190:46227)
0809 16:39:19.885860 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.885863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.886063 (+ 200us) server_negotiation.cc:407] Connection header received
0809 16:39:19.887076 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.887077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.887090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.887158 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:19.887845 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.888523 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.889639 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.889825 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.889887 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.892237 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:19.892239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:19.892239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:19.892247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:19.893862 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.894345 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.894346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.894347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.894413 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.894741 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.894742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.894742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.894877 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:19.894966 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.895317 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.895387 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:19.895714 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.885746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46227 (local address 127.28.62.131:32853)
0809 16:39:19.885939 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.885981 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.886911 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.887220 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.887224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.887499 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:19.887753 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.887758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.888636 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.888637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.889543 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.889547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.889952 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.889953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.890063 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.890602 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:19.890610 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:19.892133 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:19.893968 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.893976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.893978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.894270 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.894502 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:19.894502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:19.894503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:19.894668 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:19.894956 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:19.894963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:19.895172 (+ 209us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.895289 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.895441 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260809 16:39:19.896257 30125 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46227
I20260809 16:39:19.896374 30125 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:19.896575 30125 heartbeater.cc:511] Master 127.28.62.190:46227 requested a full tablet report, sending...
I20260809 16:39:19.896938 29592 ts_manager.cc:194] Registered new tserver with Master: 7a7db8e6b83c430185e6e71c174e4a22 (127.28.62.131:33655)
I20260809 16:39:19.897321 29592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:32853
I20260809 16:39:19.906512 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:19.914703 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.908319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.28.62.190:46227)
0809 16:39:19.908400 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.908402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.908486 (+ 84us) server_negotiation.cc:407] Connection header received
0809 16:39:19.908571 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.908572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.908582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.908629 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:19.909203 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.909693 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.910515 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.910727 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.910809 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.912032 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:19.912033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:19.912034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:19.912051 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:19.913388 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.913787 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.913788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.913789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.913855 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:19.914057 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:19.914058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:19.914058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:19.914258 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:19.914367 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.914507 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.914597 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 16:39:19.916923 29592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:19.918581 29592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:19.925655 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.28.62.131:33655)
0809 16:39:19.921718 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.921722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.921731 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:19.921771 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.921773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.921789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.922038 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:19.922139 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.922817 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.924360 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.925138 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.925218 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.925288 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.925532 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.925571 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":100,"threads_started":1}
I20260809 16:39:19.925658 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:45059 (local address 127.0.0.1:52514)
0809 16:39:19.921312 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.921354 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.921423 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.921833 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.921834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.921836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:19.921938 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.921941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.923343 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.923343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.924391 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.924393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.925329 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.925329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.925418 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.925434 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.925503 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.925566 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":68,"threads_started":1}
I20260809 16:39:19.925884 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:33655 (local address 127.0.0.1:41454)
0809 16:39:19.921518 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.921527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.921589 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.921887 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.921888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.921890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:19.921954 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.921956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.922930 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.922931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.924280 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.924283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.925326 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.925327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.925415 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.925434 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.925514 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.925559 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":7,"thread_start_us":197,"threads_started":1}
I20260809 16:39:19.926121 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.28.62.130:45059)
0809 16:39:19.921616 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.921620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.921631 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:19.921701 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.921703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.921720 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.921834 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:19.922039 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.923232 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.924484 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.925166 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.925303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.925357 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.925617 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.925659 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260809 16:39:19.926452 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38237 (local address 127.0.0.1:57906)
0809 16:39:19.921396 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:19.921407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:19.921462 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:19.921801 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:19.921804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:19.921807 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:19.921959 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.921962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.924481 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.924482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.925131 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:19.925132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.926128 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.926129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:19.926283 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.926287 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:19.926350 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 16:39:19.926389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":2,"thread_start_us":200,"threads_started":1}
I20260809 16:39:19.926730 30055 tablet_service.cc:1467] Processing CreateTablet for tablet e5c44d367ab645f79ec1038eceea5f19 (DEFAULT_TABLE table=test-table [id=c22a9a8b0e2b4996bee5527c794f14c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:19.926752 29899 tablet_service.cc:1467] Processing CreateTablet for tablet e5c44d367ab645f79ec1038eceea5f19 (DEFAULT_TABLE table=test-table [id=c22a9a8b0e2b4996bee5527c794f14c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:19.927132 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c44d367ab645f79ec1038eceea5f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:19.927130 30055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c44d367ab645f79ec1038eceea5f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:19.929689 30144 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Bootstrap starting.
I20260809 16:39:19.929728 30145 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Bootstrap starting.
I20260809 16:39:19.930123 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:19.921414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.28.62.129:38237)
0809 16:39:19.921580 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:19.921585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:19.921595 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:19.921633 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:19.921635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:19.921655 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:19.922049 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:19.922147 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.923226 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.925220 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:19.925989 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:19.926108 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:19.926177 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:19.929956 (+ 3779us) server_negotiation.cc:299] Negotiation successful
0809 16:39:19.930013 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260809 16:39:19.930430 30145 tablet_bootstrap.cc:654] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:19.930819 30144 tablet_bootstrap.cc:654] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:19.930910 29739 tablet_service.cc:1467] Processing CreateTablet for tablet e5c44d367ab645f79ec1038eceea5f19 (DEFAULT_TABLE table=test-table [id=c22a9a8b0e2b4996bee5527c794f14c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:19.931046 30145 log.cc:824] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:19.931319 29739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c44d367ab645f79ec1038eceea5f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:19.931449 30144 log.cc:824] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:19.931900 30145 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: No bootstrap required, opened a new log
I20260809 16:39:19.931959 30145 ts_tablet_manager.cc:1397] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:19.932330 30144 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: No bootstrap required, opened a new log
I20260809 16:39:19.932401 30144 ts_tablet_manager.cc:1397] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:39:19.934067 30148 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Bootstrap starting.
I20260809 16:39:19.934008 30145 raft_consensus.cc:348] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.934216 30145 raft_consensus.cc:374] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:19.934260 30145 raft_consensus.cc:729] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7db8e6b83c430185e6e71c174e4a22, State: Initialized, Role: FOLLOWER
I20260809 16:39:19.934402 30145 consensus_queue.cc:260] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.934383 30144 raft_consensus.cc:348] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.934566 30144 raft_consensus.cc:374] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:19.934623 30144 raft_consensus.cc:729] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2de714cedd4162829e5982578661ff, State: Initialized, Role: FOLLOWER
I20260809 16:39:19.934711 30145 ts_tablet_manager.cc:1428] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:19.934759 30125 heartbeater.cc:503] Master 127.28.62.190:46227 was elected leader, sending a full tablet report...
I20260809 16:39:19.934777 30144 consensus_queue.cc:260] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.935128 30148 tablet_bootstrap.cc:654] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:19.935123 29978 heartbeater.cc:503] Master 127.28.62.190:46227 was elected leader, sending a full tablet report...
I20260809 16:39:19.935115 30144 ts_tablet_manager.cc:1428] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:19.935739 30148 log.cc:824] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:19.936491 30148 tablet_bootstrap.cc:492] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: No bootstrap required, opened a new log
I20260809 16:39:19.936573 30148 ts_tablet_manager.cc:1397] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260809 16:39:19.937887 30148 raft_consensus.cc:348] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.938037 30148 raft_consensus.cc:374] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:19.938083 30148 raft_consensus.cc:729] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0670ee246ee8480e83f42ffc6739fe16, State: Initialized, Role: FOLLOWER
I20260809 16:39:19.938191 30148 consensus_queue.cc:260] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:19.938506 30148 ts_tablet_manager.cc:1428] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:39:19.938576 29817 heartbeater.cc:503] Master 127.28.62.190:46227 was elected leader, sending a full tablet report...
W20260809 16:39:19.973189 29979 tablet.cc:2369] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:39:20.063712 29818 tablet.cc:2369] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:39:20.135512 30126 tablet.cc:2369] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:20.325414 30151 raft_consensus.cc:482] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:20.325557 30151 raft_consensus.cc:504] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:20.325982 30151 leader_election.cc:288] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7db8e6b83c430185e6e71c174e4a22 (127.28.62.131:33655), 0670ee246ee8480e83f42ffc6739fe16 (127.28.62.129:38237)
I20260809 16:39:20.330101 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.326328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:38947 (local address 127.28.62.129:38237)
0809 16:39:20.326376 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:20.326379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:20.326445 (+ 66us) server_negotiation.cc:407] Connection header received
0809 16:39:20.326516 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:20.326517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:20.326527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:20.326609 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:20.326872 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.327678 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329366 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.329742 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329830 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.329840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:20.329995 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 16:39:20.330032 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:20.330117 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.326227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38237 (local address 127.28.62.130:38947)
0809 16:39:20.326371 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:20.326385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:20.326448 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:20.326656 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:20.326657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:20.326660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:20.326779 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:20.326782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.327771 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.327772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:20.329250 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:20.329252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.329831 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:20.329912 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.329916 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:20.329976 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 16:39:20.330024 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":120,"thread_start_us":76,"threads_started":1}
I20260809 16:39:20.330163 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.326374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:44217 (local address 127.28.62.131:33655)
0809 16:39:20.326418 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:20.326420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:20.326434 (+ 14us) server_negotiation.cc:407] Connection header received
0809 16:39:20.326522 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:20.326523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:20.326533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:20.326609 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:20.326871 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.327654 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329339 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.329771 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329882 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.329891 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:20.330051 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 16:39:20.330096 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:20.330410 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.326240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:33655 (local address 127.28.62.130:44217)
0809 16:39:20.326375 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:20.326385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:20.326448 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:20.326656 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:20.326657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:20.326661 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:20.326779 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:20.326781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.327772 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.327772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:20.329216 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:20.329220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.329885 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.329887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:20.329964 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.329968 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:20.330030 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 16:39:20.330138 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":95,"thread_start_us":65,"threads_started":1}
I20260809 16:39:20.330518 29762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c44d367ab645f79ec1038eceea5f19" candidate_uuid: "7b2de714cedd4162829e5982578661ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0670ee246ee8480e83f42ffc6739fe16" is_pre_election: true
I20260809 16:39:20.330787 29762 raft_consensus.cc:2454] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2de714cedd4162829e5982578661ff in term 0.
I20260809 16:39:20.330775 30075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c44d367ab645f79ec1038eceea5f19" candidate_uuid: "7b2de714cedd4162829e5982578661ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7db8e6b83c430185e6e71c174e4a22" is_pre_election: true
I20260809 16:39:20.330906 30075 raft_consensus.cc:2454] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2de714cedd4162829e5982578661ff in term 0.
I20260809 16:39:20.331107 29850 leader_election.cc:302] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0670ee246ee8480e83f42ffc6739fe16, 7b2de714cedd4162829e5982578661ff; no voters:
I20260809 16:39:20.331278 30151 raft_consensus.cc:2781] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:20.331368 30151 raft_consensus.cc:482] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:20.331410 30151 raft_consensus.cc:3037] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:20.332139 30151 raft_consensus.cc:504] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:20.332301 30151 leader_election.cc:288] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [CANDIDATE]: Term 1 election: Requested vote from peers 7a7db8e6b83c430185e6e71c174e4a22 (127.28.62.131:33655), 0670ee246ee8480e83f42ffc6739fe16 (127.28.62.129:38237)
I20260809 16:39:20.332458 30075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c44d367ab645f79ec1038eceea5f19" candidate_uuid: "7b2de714cedd4162829e5982578661ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7db8e6b83c430185e6e71c174e4a22"
I20260809 16:39:20.332469 29762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c44d367ab645f79ec1038eceea5f19" candidate_uuid: "7b2de714cedd4162829e5982578661ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0670ee246ee8480e83f42ffc6739fe16"
I20260809 16:39:20.332559 30075 raft_consensus.cc:3037] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:20.332561 29762 raft_consensus.cc:3037] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:20.333297 30075 raft_consensus.cc:2454] T e5c44d367ab645f79ec1038eceea5f19 P 7a7db8e6b83c430185e6e71c174e4a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2de714cedd4162829e5982578661ff in term 1.
I20260809 16:39:20.333317 29762 raft_consensus.cc:2454] T e5c44d367ab645f79ec1038eceea5f19 P 0670ee246ee8480e83f42ffc6739fe16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2de714cedd4162829e5982578661ff in term 1.
I20260809 16:39:20.333500 29850 leader_election.cc:302] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0670ee246ee8480e83f42ffc6739fe16, 7b2de714cedd4162829e5982578661ff; no voters:
I20260809 16:39:20.333593 30151 raft_consensus.cc:2781] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:20.333745 30151 raft_consensus.cc:686] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [term 1 LEADER]: Becoming Leader. State: Replica: 7b2de714cedd4162829e5982578661ff, State: Running, Role: LEADER
I20260809 16:39:20.333858 30151 consensus_queue.cc:237] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } }
I20260809 16:39:20.334707 29592 catalog_manager.cc:5495] T e5c44d367ab645f79ec1038eceea5f19 P 7b2de714cedd4162829e5982578661ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b2de714cedd4162829e5982578661ff (127.28.62.130). New cstate: current_term: 1 leader_uuid: "7b2de714cedd4162829e5982578661ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2de714cedd4162829e5982578661ff" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7db8e6b83c430185e6e71c174e4a22" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0670ee246ee8480e83f42ffc6739fe16" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38237 } health_report { overall_health: UNKNOWN } } }
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:20.372947 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.364703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.28.62.129:38237)
0809 16:39:20.364770 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:20.364773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:20.364899 (+ 126us) server_negotiation.cc:407] Connection header received
0809 16:39:20.364962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:20.364964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:20.364977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:20.365048 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:20.365954 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.366538 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.367383 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.367603 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.367663 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.370228 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:20.370239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:20.370239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:20.370246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:20.371795 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.372185 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.372188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.372188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.372250 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.372461 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.372461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.372462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.372610 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:20.372726 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:20.372786 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 16:39:20.372850 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293560, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293560, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:20.398427 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.390688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.28.62.129:38237)
0809 16:39:20.390791 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:20.390794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:20.390886 (+ 92us) server_negotiation.cc:407] Connection header received
0809 16:39:20.390990 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:20.390991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:20.391006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:20.391111 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:20.391702 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.392213 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.392947 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.393141 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.393216 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.395533 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:20.395535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:20.395535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:20.395543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:20.397035 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.397501 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.397502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.397503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.397566 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.397788 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.397789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.397789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.397899 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:20.398008 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:20.398215 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 16:39:20.398307 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[29538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:20.422778 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:20.415375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.28.62.129:38237)
0809 16:39:20.415443 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:20.415446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:20.415506 (+ 60us) server_negotiation.cc:407] Connection header received
0809 16:39:20.415604 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:20.415605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:20.415616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:20.415671 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:20.416233 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.416724 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.417499 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:20.417707 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:20.417789 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:20.420166 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:20.420167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:20.420168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:20.420177 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:20.421565 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.421895 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.421896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.421896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.421945 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:20.422177 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:20.422179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:20.422179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:20.422355 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:20.422448 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:20.422618 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 16:39:20.422687 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 16:39:20.423431 29714 tablet_service.cc:2908] Rejecting scan request for tablet e5c44d367ab645f79ec1038eceea5f19: Uninitialized: clean time has not yet been initialized
I20260809 16:39:20.424351 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29661
I20260809 16:39:20.436309 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29823
I20260809 16:39:20.442683 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29984
I20260809 16:39:20.450495 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 29553
2026-08-09T16:39:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2511 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[30176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[30176](info): set up 2 sockets
Aug 09 16:39:20 dist-test-slave-wl2h krb5kdc[30176](info): commencing operation
krb5kdc: starting...
W20260809 16:39:21.579407 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:21 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:21.707355 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:37773
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:37773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:21.790656 30191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:21.790881 30191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:21.790904 30191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:21.792026 30191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:21.792081 30191 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 16:39:21.792145 30191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:21.792176 30191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:21.792197 30191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:21.792220 30191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:21.793723 30191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:37773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:37773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:21.793929 30191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:21.794107 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:21.798964 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:21.798971 30199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:21.799084 30196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:21.799268 30191 server_base.cc:1034] running on GCE node
I20260809 16:39:21.799515 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:21.799767 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:21.800908 30191 hybrid_clock.cc:648] HybridClock initialized: now 1786293561800897 us; error 30 us; skew 500 ppm
Aug 09 16:39:21 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:21.808964 30191 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:21.809477 30191 webserver.cc:460] Webserver started at http://127.28.62.190:39365/ using document root <none> and password file <none>
I20260809 16:39:21.809715 30191 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:21.809762 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:21.809930 30191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:21.811091 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "f48891ef8d7b42b7b8102a2359e861eb"
format_stamp: "Formatted at 2026-08-09 16:39:21 on dist-test-slave-wl2h"
server_key: "d6233717323e51ac882a566ad30606bd"
server_key_iv: "90d697131db09c6e0c3bc62b1ea629a9"
server_key_version: "encryptionkey@0"
I20260809 16:39:21.811471 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "f48891ef8d7b42b7b8102a2359e861eb"
format_stamp: "Formatted at 2026-08-09 16:39:21 on dist-test-slave-wl2h"
server_key: "d6233717323e51ac882a566ad30606bd"
server_key_iv: "90d697131db09c6e0c3bc62b1ea629a9"
server_key_version: "encryptionkey@0"
I20260809 16:39:21.812988 30191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:39:21.813839 30206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:21.814046 30191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:21.814159 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "f48891ef8d7b42b7b8102a2359e861eb"
format_stamp: "Formatted at 2026-08-09 16:39:21 on dist-test-slave-wl2h"
server_key: "d6233717323e51ac882a566ad30606bd"
server_key_iv: "90d697131db09c6e0c3bc62b1ea629a9"
server_key_version: "encryptionkey@0"
I20260809 16:39:21.814256 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:21.824571 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:21.826639 30191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:21.826818 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:21.832197 30191 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:37773
I20260809 16:39:21.832255 30279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:37773 every 8 connection(s)
I20260809 16:39:21.832594 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:21.832855 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 30191
I20260809 16:39:21.832967 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:21.833145 28922 external_mini_cluster.cc:1428] Setting key fc091d3d18147b86a2007c40f92c2c97
I20260809 16:39:21.833395 30280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:21.836337 30280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Bootstrap starting.
I20260809 16:39:21.836971 30280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:21.837276 30280 log.cc:824] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:21.837962 30280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: No bootstrap required, opened a new log
Aug 09 16:39:21 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:21.840381 30280 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } }
I20260809 16:39:21.840521 30280 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:21.840544 30280 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48891ef8d7b42b7b8102a2359e861eb, State: Initialized, Role: FOLLOWER
I20260809 16:39:21.840687 30280 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } }
I20260809 16:39:21.840755 30280 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:21.840776 30280 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:21.840838 30280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:21.841396 30280 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } }
I20260809 16:39:21.841506 30280 leader_election.cc:302] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48891ef8d7b42b7b8102a2359e861eb; no voters:
I20260809 16:39:21.841684 30280 leader_election.cc:288] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:21.841717 30286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:21.841826 30286 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [term 1 LEADER]: Becoming Leader. State: Replica: f48891ef8d7b42b7b8102a2359e861eb, State: Running, Role: LEADER
I20260809 16:39:21.841964 30280 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:21.841967 30286 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } }
I20260809 16:39:21.842403 30286 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48891ef8d7b42b7b8102a2359e861eb. Latest consensus state: current_term: 1 leader_uuid: "f48891ef8d7b42b7b8102a2359e861eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } } }
I20260809 16:39:21.842411 30288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48891ef8d7b42b7b8102a2359e861eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48891ef8d7b42b7b8102a2359e861eb" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37773 } } }
I20260809 16:39:21.842624 30288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:21.842796 30286 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:21.843036 30297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:21.843624 30297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:21.844398 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:21.833690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.28.62.190:37773)
0809 16:39:21.833847 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:21.833856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:21.833952 (+ 96us) server_negotiation.cc:407] Connection header received
0809 16:39:21.834436 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:21.834443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:21.834488 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:21.834650 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:21.835340 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:21.836221 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:21.837030 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:21.837524 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:21.837657 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:21.840346 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:21.840349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:21.840349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:21.840359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:21.842408 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:21.842848 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:21.842852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:21.842852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:21.842967 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:21.843296 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:21.843297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:21.843297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:21.843646 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:21.843805 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:21.844056 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 16:39:21.844195 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260809 16:39:21.845299 30297 catalog_manager.cc:1347] Generated new cluster ID: f8f982111b8a414ba16e3e3db5376bb6
I20260809 16:39:21.845367 30297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:21.867647 30297 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:21.868217 30297 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:21.875196 30297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Generated new TSK 0
I20260809 16:39:21.875422 30297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:21.906764 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:37773
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:21.991953 30311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:21.992192 30311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:21.992218 30311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:21.993373 30311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:21.993415 30311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:21.993435 30311 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:21.993453 30311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:21.994891 30311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:37773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:21.995240 30311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:21.995492 30311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:22.000766 30317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.000707 30320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.000735 30318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:22.000885 30311 server_base.cc:1034] running on GCE node
I20260809 16:39:22.001199 30311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:22.001454 30311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:22.002642 30311 hybrid_clock.cc:648] HybridClock initialized: now 1786293562002637 us; error 39 us; skew 500 ppm
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:22.005287 30311 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:22.005733 30311 webserver.cc:460] Webserver started at http://127.28.62.129:38815/ using document root <none> and password file <none>
I20260809 16:39:22.005959 30311 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:22.006024 30311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:22.006173 30311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:22.007305 30311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "c7fe3b628fda4145be4b29dd4d8466d5"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "9a35b198e04f3f71de1f8d3755e3b50e"
server_key_iv: "6d0ad486404626d8bd6febc7789b93b0"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.007669 30311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "c7fe3b628fda4145be4b29dd4d8466d5"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "9a35b198e04f3f71de1f8d3755e3b50e"
server_key_iv: "6d0ad486404626d8bd6febc7789b93b0"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.009074 30311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:22.009830 30327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.009999 30311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:22.010100 30311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "c7fe3b628fda4145be4b29dd4d8466d5"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "9a35b198e04f3f71de1f8d3755e3b50e"
server_key_iv: "6d0ad486404626d8bd6febc7789b93b0"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.010175 30311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:22.039883 30311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:22.042109 30311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:22.042364 30311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:22.043154 30311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:22.043216 30311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.043262 30311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:22.043291 30311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:22.050794 30311 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:36025
I20260809 16:39:22.050865 30458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:36025 every 8 connection(s)
I20260809 16:39:22.051236 30311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:22.052804 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 30311
I20260809 16:39:22.052888 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:22.053097 28922 external_mini_cluster.cc:1428] Setting key b01f9bb2ca65155bf435a71d7fc99f24
I20260809 16:39:22.053807 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.043838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:51021 (local address 127.28.62.190:37773)
0809 16:39:22.043907 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.043910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.044063 (+ 153us) server_negotiation.cc:407] Connection header received
0809 16:39:22.044580 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.044581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.044596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.044678 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.045634 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.046527 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.047603 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.047967 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.048053 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.050402 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.050404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.050404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.050411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.052072 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.052585 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.052586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.052586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.052652 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.053007 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.053009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.053009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.053171 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.053246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.053595 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.053695 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:39:22.053979 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.043747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.129:51021)
0809 16:39:22.043926 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.043968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.044459 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.044739 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.044742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.045212 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.045520 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.045526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.046628 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.046629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.047529 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.047531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.048075 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.048076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.048210 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.048748 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.048757 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.050311 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.052181 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.052189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.052191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.052492 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.052738 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.052739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.052739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.052869 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.053279 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.053286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.053424 (+ 138us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.053554 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.053729 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260809 16:39:22.055653 30460 heartbeater.cc:347] Connected to a master server at 127.28.62.190:37773
I20260809 16:39:22.055804 30460 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:22.055992 30460 heartbeater.cc:511] Master 127.28.62.190:37773 requested a full tablet report, sending...
I20260809 16:39:22.056553 30227 ts_manager.cc:194] Registered new tserver with Master: c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025)
I20260809 16:39:22.057444 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:51021
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:39:22.063567 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.056881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:51565 (local address 127.28.62.190:37773)
0809 16:39:22.056924 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.056926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.057133 (+ 207us) server_negotiation.cc:407] Connection header received
0809 16:39:22.057184 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.057185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.057195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.057249 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.057624 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.058353 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.059493 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.059684 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.059740 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.060807 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.060808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.060808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.060814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.062245 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.062669 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.062670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.062670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.062726 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.062930 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.062930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.062930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.063090 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.063222 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.063388 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.063472 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:22.063571 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.056862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.129:51565)
0809 16:39:22.057055 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.057067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.057132 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.057254 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.057254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.057478 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.057558 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.057560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.058443 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.058444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.059409 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.059412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.059761 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.059762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.059851 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.060283 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.060288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.060707 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.062329 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.062330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.062331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.062569 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.062798 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.062799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.062800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.062868 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.063219 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.063220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.063271 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.063361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.063470 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:22.083154 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:37773
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:22.171443 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:22.171659 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:22.171686 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:22.172848 30475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:22.172887 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:22.172909 30475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:22.172926 30475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:22.174350 30475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:37773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:22.174554 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:22.174746 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:22.180486 30482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.180480 30483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.180477 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:22.180830 30475 server_base.cc:1034] running on GCE node
I20260809 16:39:22.181032 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:22.181298 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:22.182444 30475 hybrid_clock.cc:648] HybridClock initialized: now 1786293562182425 us; error 39 us; skew 500 ppm
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:22.185133 30475 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:22.185642 30475 webserver.cc:460] Webserver started at http://127.28.62.130:46421/ using document root <none> and password file <none>
I20260809 16:39:22.185873 30475 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:22.185938 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:22.186085 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:22.187224 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "44a13e281ccb4c3caf3347672793b015"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "04630fecacafe5e50d2d0d63ae4e2fdd"
server_key_iv: "8c41d56f4a3dd58f7ebabb390f5860c4"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.187580 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "44a13e281ccb4c3caf3347672793b015"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "04630fecacafe5e50d2d0d63ae4e2fdd"
server_key_iv: "8c41d56f4a3dd58f7ebabb390f5860c4"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.189002 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:22.189801 30493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.190004 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:22.190109 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "44a13e281ccb4c3caf3347672793b015"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "04630fecacafe5e50d2d0d63ae4e2fdd"
server_key_iv: "8c41d56f4a3dd58f7ebabb390f5860c4"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.190181 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:22.201769 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:22.204170 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:22.204341 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:22.204999 30475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:22.205053 30475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.205087 30475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:22.205108 30475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:22.213076 30475 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:40499
I20260809 16:39:22.213442 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:22.213968 30626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:40499 every 8 connection(s)
I20260809 16:39:22.216382 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.205482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:54015 (local address 127.28.62.190:37773)
0809 16:39:22.205553 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.205555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.205831 (+ 276us) server_negotiation.cc:407] Connection header received
0809 16:39:22.206592 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.206594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.206609 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.206695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.207668 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.208245 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.209945 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.210197 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.210283 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.213011 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.213013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.213013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.213022 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.214787 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.215294 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.215296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.215296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.215372 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.215649 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.215650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.215650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.215780 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.215860 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.216154 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.216252 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:22.216590 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.205457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.130:54015)
0809 16:39:22.205668 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.205724 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.206463 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.206787 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.206792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.207232 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.207549 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.207557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.208362 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.208364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.209848 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.209852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.210323 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.210324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.210461 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.211060 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.211070 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.212897 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.214881 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.214889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.214891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.215206 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.215446 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.215447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.215447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.215570 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.215852 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.215859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.216009 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.216116 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.216291 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260809 16:39:22.217067 30627 heartbeater.cc:347] Connected to a master server at 127.28.62.190:37773
I20260809 16:39:22.217155 30627 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:22.217346 30627 heartbeater.cc:511] Master 127.28.62.190:37773 requested a full tablet report, sending...
I20260809 16:39:22.217814 30226 ts_manager.cc:194] Registered new tserver with Master: 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499)
I20260809 16:39:22.218374 30226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:54015
I20260809 16:39:22.219034 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 30475
I20260809 16:39:22.219132 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:22.219323 28922 external_mini_cluster.cc:1428] Setting key 2e4925c68685cfcf27072749846405f7
I20260809 16:39:22.224596 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.218311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60653 (local address 127.28.62.190:37773)
0809 16:39:22.218347 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.218349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.218416 (+ 67us) server_negotiation.cc:407] Connection header received
0809 16:39:22.218492 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.218492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.218503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.218563 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.218987 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.219529 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.220449 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.220760 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.220837 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.221759 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.221761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.221761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.221769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.223363 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.223758 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.223759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.223759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.223811 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.224007 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.224008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.224008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.224144 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.224228 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.224405 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.224515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:39:22.224649 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.218223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.130:60653)
0809 16:39:22.218356 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.218366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.218428 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.218571 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.218572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.218833 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.218897 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.218899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.219637 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.219638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.220366 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.220369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.220853 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.220854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.220912 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.221363 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.221368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.221668 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.223446 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.223447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.223447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.223683 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.223874 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.223874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.223875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.223946 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.224244 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.224245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.224297 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.224377 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.224493 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:22.249894 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:37773
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:22.333117 30639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:22.333317 30639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:22.333343 30639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:22.334429 30639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:22.334462 30639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:22.334483 30639 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:22.334501 30639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:22.335872 30639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:37773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:22.336051 30639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:22.336222 30639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:22.341172 30646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.341183 30645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:22.341236 30649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:22.341763 30639 server_base.cc:1034] running on GCE node
I20260809 16:39:22.341966 30639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:22.342239 30639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:22.343446 30639 hybrid_clock.cc:648] HybridClock initialized: now 1786293562343429 us; error 25 us; skew 500 ppm
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:22.346084 30639 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:22.346499 30639 webserver.cc:460] Webserver started at http://127.28.62.131:38061/ using document root <none> and password file <none>
I20260809 16:39:22.346747 30639 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:22.346813 30639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:22.346958 30639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:22.348078 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "44627453b2b949ffa978c541bddb2edc"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "f84d147afc0d3ecafaf10153aca5397e"
server_key_iv: "e47c5cf9fcb3e0cc6a42a4b192debeac"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.348456 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "44627453b2b949ffa978c541bddb2edc"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "f84d147afc0d3ecafaf10153aca5397e"
server_key_iv: "e47c5cf9fcb3e0cc6a42a4b192debeac"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.349896 30639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:22.350649 30656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.350869 30639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:22.350976 30639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "44627453b2b949ffa978c541bddb2edc"
format_stamp: "Formatted at 2026-08-09 16:39:22 on dist-test-slave-wl2h"
server_key: "f84d147afc0d3ecafaf10153aca5397e"
server_key_iv: "e47c5cf9fcb3e0cc6a42a4b192debeac"
server_key_version: "encryptionkey@0"
I20260809 16:39:22.351086 30639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:22.367970 30639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:22.370119 30639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:22.370350 30639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:22.370906 30639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:22.370970 30639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.371016 30639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:22.371076 30639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:22.378970 30639 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36819
I20260809 16:39:22.379081 30788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36819 every 8 connection(s)
I20260809 16:39:22.379400 30639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:22.381880 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.371471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:48875 (local address 127.28.62.190:37773)
0809 16:39:22.371543 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.371546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.371808 (+ 262us) server_negotiation.cc:407] Connection header received
0809 16:39:22.372559 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.372560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.372570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.372604 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.373533 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.374455 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.376203 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.376553 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.376632 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.379187 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.379189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.379190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.379198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.380609 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.380973 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.380974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.380974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.381043 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.381310 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.381310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.381310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.381414 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.381467 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.381757 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.381817 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:22.382124 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.371448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.131:48875)
0809 16:39:22.371660 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.371714 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.372446 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.372700 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.372704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.373108 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.373418 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.373425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.374555 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.374555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.376087 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.376091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.376678 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.376680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.376824 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.377359 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.377367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.379041 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.380686 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.380693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.380694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.380904 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.381142 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.381142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.381143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.381251 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.381492 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.381499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.381626 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.381732 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.381856 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260809 16:39:22.382630 30789 heartbeater.cc:347] Connected to a master server at 127.28.62.190:37773
I20260809 16:39:22.382746 30789 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:22.382942 30789 heartbeater.cc:511] Master 127.28.62.190:37773 requested a full tablet report, sending...
I20260809 16:39:22.383351 30227 ts_manager.cc:194] Registered new tserver with Master: 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819)
I20260809 16:39:22.383773 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:48875
I20260809 16:39:22.385504 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 30639
I20260809 16:39:22.385599 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:22.385784 28922 external_mini_cluster.cc:1428] Setting key d2673e50d62714e0d0db2b79868f1354
I20260809 16:39:22.386684 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:22.389905 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.383745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:39087 (local address 127.28.62.190:37773)
0809 16:39:22.383790 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.383793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.383909 (+ 116us) server_negotiation.cc:407] Connection header received
0809 16:39:22.383965 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.383965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.383976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.384039 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.384357 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.384831 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.385665 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.385940 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.386021 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.387069 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.387070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.387071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.387077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.388773 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.389237 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.389238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.389238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.389287 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.389493 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.389493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.389493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.389583 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.389627 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.389796 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.389848 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:22.390027 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.383681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.28.62.131:39087)
0809 16:39:22.383851 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.383862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.383924 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.384041 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.384041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.384221 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.384276 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.384277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.384932 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.384933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.385539 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.385552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.386053 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.386054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.386156 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.386617 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.386623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.386960 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.388859 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.388860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.388860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.389145 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.389352 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.389353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.389354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.389427 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.389638 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.389644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.389696 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.389782 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.389893 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:22.411206 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.404209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.28.62.190:37773)
0809 16:39:22.404273 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.404275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.404285 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:22.404323 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.404324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.404333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.404396 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.404967 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.405469 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.406240 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.406429 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.406498 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.408767 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.408768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.408769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.408777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.410031 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.410394 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.410395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.410396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.410452 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.410647 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.410648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.410648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.410755 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.410866 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.410960 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.411089 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:39:22.412854 30227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:22.414628 30227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:22.421386 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36025 (local address 127.0.0.1:46660)
0809 16:39:22.417269 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.417314 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.417383 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.417797 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.417800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.417802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.417871 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.417873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.419320 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.419321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.420359 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.420361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.421057 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.421143 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421157 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.421242 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.421299 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":62,"threads_started":1}
I20260809 16:39:22.421408 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.28.62.129:36025)
0809 16:39:22.417547 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.417551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.417563 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:22.417687 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.417690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.417707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.417782 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.417995 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.419234 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.420464 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.420969 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421057 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421118 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.421271 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.421309 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260809 16:39:22.421667 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36819 (local address 127.0.0.1:45520)
0809 16:39:22.417475 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.417486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.417543 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.417921 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.417922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.417924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.418002 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.418004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.419039 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.419040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.420351 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.420354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.421372 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.421442 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421446 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.421497 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.421550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260809 16:39:22.421841 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40499 (local address 127.0.0.1:55864)
0809 16:39:22.417591 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.417602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.417650 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.418293 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.418294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.418296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.418344 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.418345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.419554 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.419555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.420430 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.420433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.421374 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.421442 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421606 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.421654 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.421696 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":160,"thread_start_us":96,"threads_started":1}
I20260809 16:39:22.422104 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.28.62.130:40499)
0809 16:39:22.418123 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.418127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.418136 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:22.418174 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.418175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.418191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.418274 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.418454 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.419470 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.420532 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.421224 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421283 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421331 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.421943 (+ 612us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.421992 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":99,"threads_started":1}
I20260809 16:39:22.422147 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.417234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.28.62.131:36819)
0809 16:39:22.417669 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.417673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.417684 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:22.417724 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.417726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.417743 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.417939 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.418103 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.418937 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.420450 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.421199 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.421278 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.421332 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.421801 (+ 469us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.421846 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":91,"threads_started":1}
I20260809 16:39:22.422192 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 9951941a2ce1439797dd50dfcea93f4b (DEFAULT_TABLE table=test-table [id=d1c9203f7c62453a8e0c51f7361815a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.422616 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9951941a2ce1439797dd50dfcea93f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.422734 30552 tablet_service.cc:1467] Processing CreateTablet for tablet 9951941a2ce1439797dd50dfcea93f4b (DEFAULT_TABLE table=test-table [id=d1c9203f7c62453a8e0c51f7361815a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.423077 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9951941a2ce1439797dd50dfcea93f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.426254 30715 tablet_service.cc:1467] Processing CreateTablet for tablet 9951941a2ce1439797dd50dfcea93f4b (DEFAULT_TABLE table=test-table [id=d1c9203f7c62453a8e0c51f7361815a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.426586 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9951941a2ce1439797dd50dfcea93f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.426806 30818 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Bootstrap starting.
I20260809 16:39:22.426956 30819 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Bootstrap starting.
I20260809 16:39:22.427695 30819 tablet_bootstrap.cc:654] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.428066 30819 log.cc:824] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:22.428808 30819 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: No bootstrap required, opened a new log
I20260809 16:39:22.428892 30819 ts_tablet_manager.cc:1397] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 16:39:22.428925 30818 tablet_bootstrap.cc:654] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.429281 30818 log.cc:824] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:22.430084 30818 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: No bootstrap required, opened a new log
I20260809 16:39:22.430245 30822 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Bootstrap starting.
I20260809 16:39:22.430279 30818 ts_tablet_manager.cc:1397] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 16:39:22.430948 30819 raft_consensus.cc:348] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.431155 30819 raft_consensus.cc:374] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.431208 30819 raft_consensus.cc:729] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a13e281ccb4c3caf3347672793b015, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.431329 30822 tablet_bootstrap.cc:654] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.431357 30819 consensus_queue.cc:260] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.431674 30819 ts_tablet_manager.cc:1428] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 16:39:22.431676 30822 log.cc:824] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:22.431802 30627 heartbeater.cc:503] Master 127.28.62.190:37773 was elected leader, sending a full tablet report...
I20260809 16:39:22.432039 30818 raft_consensus.cc:348] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.432219 30818 raft_consensus.cc:374] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.432264 30818 raft_consensus.cc:729] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fe3b628fda4145be4b29dd4d8466d5, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.432377 30818 consensus_queue.cc:260] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.432456 30822 tablet_bootstrap.cc:492] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: No bootstrap required, opened a new log
I20260809 16:39:22.432518 30822 ts_tablet_manager.cc:1397] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:22.432706 30818 ts_tablet_manager.cc:1428] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:22.432739 30460 heartbeater.cc:503] Master 127.28.62.190:37773 was elected leader, sending a full tablet report...
I20260809 16:39:22.434577 30822 raft_consensus.cc:348] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.434691 30822 raft_consensus.cc:374] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.434711 30822 raft_consensus.cc:729] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44627453b2b949ffa978c541bddb2edc, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.434800 30822 consensus_queue.cc:260] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.435003 30822 ts_tablet_manager.cc:1428] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:22.435254 30789 heartbeater.cc:503] Master 127.28.62.190:37773 was elected leader, sending a full tablet report...
W20260809 16:39:22.464210 30628 tablet.cc:2369] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:22.505025 30827 raft_consensus.cc:482] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:22.505206 30827 raft_consensus.cc:504] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.505616 30827 leader_election.cc:288] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819), 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499)
I20260809 16:39:22.509759 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.505924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:44833 (local address 127.28.62.130:40499)
0809 16:39:22.505972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.505974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.506118 (+ 144us) server_negotiation.cc:407] Connection header received
0809 16:39:22.506181 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.506182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.506192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.506235 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.506444 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.507232 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.508781 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.509360 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.509439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.509453 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.509633 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.509671 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:22.509747 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.505878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:60497 (local address 127.28.62.131:36819)
0809 16:39:22.505945 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.505948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.505959 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:22.506051 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.506052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.506060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.506129 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.506444 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.507233 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.508782 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.509360 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.509439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.509453 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.509633 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.509674 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 16:39:22.509822 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.505819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36819 (local address 127.28.62.129:60497)
0809 16:39:22.505892 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.505905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.505954 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.506161 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.506163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.506164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.506259 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.506261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.507326 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.507326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.508689 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.508692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.509452 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.509453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.509555 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.509557 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.509616 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.509703 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":110}
I20260809 16:39:22.510002 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.505839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40499 (local address 127.28.62.129:44833)
0809 16:39:22.506033 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.506041 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.506098 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.506260 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.506261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.506262 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.506327 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.506328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.507326 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.507326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.508692 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.508693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.509452 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.509453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.509552 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.509557 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.509616 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.509695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":98,"thread_start_us":123,"threads_started":1}
I20260809 16:39:22.510187 30727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9951941a2ce1439797dd50dfcea93f4b" candidate_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44627453b2b949ffa978c541bddb2edc" is_pre_election: true
I20260809 16:39:22.510327 30573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9951941a2ce1439797dd50dfcea93f4b" candidate_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a13e281ccb4c3caf3347672793b015" is_pre_election: true
I20260809 16:39:22.510406 30727 raft_consensus.cc:2454] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fe3b628fda4145be4b29dd4d8466d5 in term 0.
I20260809 16:39:22.510462 30573 raft_consensus.cc:2454] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fe3b628fda4145be4b29dd4d8466d5 in term 0.
I20260809 16:39:22.510666 30329 leader_election.cc:302] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44627453b2b949ffa978c541bddb2edc, c7fe3b628fda4145be4b29dd4d8466d5; no voters:
I20260809 16:39:22.510815 30827 raft_consensus.cc:2781] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:22.510874 30827 raft_consensus.cc:482] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:22.510896 30827 raft_consensus.cc:3037] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.511650 30827 raft_consensus.cc:504] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.511797 30827 leader_election.cc:288] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [CANDIDATE]: Term 1 election: Requested vote from peers 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819), 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499)
I20260809 16:39:22.511950 30727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9951941a2ce1439797dd50dfcea93f4b" candidate_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44627453b2b949ffa978c541bddb2edc"
I20260809 16:39:22.511976 30573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9951941a2ce1439797dd50dfcea93f4b" candidate_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a13e281ccb4c3caf3347672793b015"
I20260809 16:39:22.512036 30727 raft_consensus.cc:3037] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.512053 30573 raft_consensus.cc:3037] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.512972 30727 raft_consensus.cc:2454] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fe3b628fda4145be4b29dd4d8466d5 in term 1.
I20260809 16:39:22.513001 30573 raft_consensus.cc:2454] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fe3b628fda4145be4b29dd4d8466d5 in term 1.
I20260809 16:39:22.513154 30328 leader_election.cc:302] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a13e281ccb4c3caf3347672793b015, c7fe3b628fda4145be4b29dd4d8466d5; no voters:
I20260809 16:39:22.513329 30827 raft_consensus.cc:2781] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:22.513479 30827 raft_consensus.cc:686] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 LEADER]: Becoming Leader. State: Replica: c7fe3b628fda4145be4b29dd4d8466d5, State: Running, Role: LEADER
I20260809 16:39:22.513608 30827 consensus_queue.cc:237] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } }
I20260809 16:39:22.514477 30227 catalog_manager.cc:5495] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129). New cstate: current_term: 1 leader_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:22.542850 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.540015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.28.62.129:36025)
0809 16:39:22.540079 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.540081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.540089 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:22.540126 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.540127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.540144 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.540200 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:22.540355 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.540816 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.541675 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.541859 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.541952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.542194 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.542288 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.542348 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.542521 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.542544 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 16:39:22.544922 30727 raft_consensus.cc:1261] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Refusing update from remote peer c7fe3b628fda4145be4b29dd4d8466d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.544922 30573 raft_consensus.cc:1261] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Refusing update from remote peer c7fe3b628fda4145be4b29dd4d8466d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.545326 30827 consensus_queue.cc:1035] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.545506 30830 consensus_queue.cc:1035] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.547808 30835 mvcc.cc:205] Tried to move back new op lower bound from 7316658432180449280 to 7316658432056455168. Current Snapshot: MvccSnapshot[applied={T|T < 7316658432180449280}]
I20260809 16:39:22.548504 30837 mvcc.cc:205] Tried to move back new op lower bound from 7316658432180449280 to 7316658432056455168. Current Snapshot: MvccSnapshot[applied={T|T < 7316658432180449280}]
I20260809 16:39:22.548977 30836 mvcc.cc:205] Tried to move back new op lower bound from 7316658432180449280 to 7316658432056455168. Current Snapshot: MvccSnapshot[applied={T|T < 7316658432180449280 or (T in {7316658432180449280})}]
I20260809 16:39:22.551754 30226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:22.551904 30226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:22.552779 30226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Sending DeleteTablet for 3 replicas of tablet 9951941a2ce1439797dd50dfcea93f4b
I20260809 16:39:22.553120 30715 tablet_service.cc:1514] Processing DeleteTablet for tablet 9951941a2ce1439797dd50dfcea93f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:45520
I20260809 16:39:22.553129 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9951941a2ce1439797dd50dfcea93f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:46660
I20260809 16:39:22.553371 30845 tablet_replica.cc:330] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: stopping tablet replica
I20260809 16:39:22.553460 30845 raft_consensus.cc:2229] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:22.553431 30226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 16:39:22.553489 30552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9951941a2ce1439797dd50dfcea93f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:55864
I20260809 16:39:22.553565 30845 raft_consensus.cc:2258] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 16:39:22.553581 30226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:22.553989 30846 tablet_replica.cc:330] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: stopping tablet replica
I20260809 16:39:22.554061 30845 ts_tablet_manager.cc:1905] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.554206 30846 raft_consensus.cc:2229] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:22.554390 30846 raft_consensus.cc:2258] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:22.554919 30846 ts_tablet_manager.cc:1905] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.555411 30552 tablet_service.cc:1467] Processing CreateTablet for tablet cd644567f6a441dea18e22d53c3df82f (DEFAULT_TABLE table=test-table [id=3edb623f25464e2ca9de96ac8ada00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.555554 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd644567f6a441dea18e22d53c3df82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.556365 30845 ts_tablet_manager.cc:1918] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:22.556442 30845 log.cc:1196] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/9951941a2ce1439797dd50dfcea93f4b
I20260809 16:39:22.556907 30845 ts_tablet_manager.cc:1939] T 9951941a2ce1439797dd50dfcea93f4b P 44627453b2b949ffa978c541bddb2edc: Deleting consensus metadata
I20260809 16:39:22.557047 30819 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Bootstrap starting.
I20260809 16:39:22.554993 30388 tablet_service.cc:1467] Processing CreateTablet for tablet cd644567f6a441dea18e22d53c3df82f (DEFAULT_TABLE table=test-table [id=3edb623f25464e2ca9de96ac8ada00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.557252 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd644567f6a441dea18e22d53c3df82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.557473 30819 tablet_bootstrap.cc:654] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.557495 30846 ts_tablet_manager.cc:1918] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:22.557555 30846 log.cc:1196] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/9951941a2ce1439797dd50dfcea93f4b
I20260809 16:39:22.557832 30846 ts_tablet_manager.cc:1939] T 9951941a2ce1439797dd50dfcea93f4b P c7fe3b628fda4145be4b29dd4d8466d5: Deleting consensus metadata
I20260809 16:39:22.558316 30819 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: No bootstrap required, opened a new log
I20260809 16:39:22.558393 30819 ts_tablet_manager.cc:1397] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:22.558501 30818 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Bootstrap starting.
I20260809 16:39:22.558573 30819 raft_consensus.cc:348] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.558677 30819 raft_consensus.cc:374] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.558702 30819 raft_consensus.cc:729] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a13e281ccb4c3caf3347672793b015, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.558753 30819 consensus_queue.cc:260] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.558856 30819 ts_tablet_manager.cc:1428] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.558975 30818 tablet_bootstrap.cc:654] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.559211 30211 catalog_manager.cc:4841] TS c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025): tablet 9951941a2ce1439797dd50dfcea93f4b (table test-table [id=d1c9203f7c62453a8e0c51f7361815a3]) successfully deleted
I20260809 16:39:22.559391 30210 catalog_manager.cc:4841] TS 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819): tablet 9951941a2ce1439797dd50dfcea93f4b (table test-table [id=d1c9203f7c62453a8e0c51f7361815a3]) successfully deleted
I20260809 16:39:22.559587 30715 tablet_service.cc:1467] Processing CreateTablet for tablet cd644567f6a441dea18e22d53c3df82f (DEFAULT_TABLE table=test-table [id=3edb623f25464e2ca9de96ac8ada00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:22.559708 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd644567f6a441dea18e22d53c3df82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.559911 30818 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: No bootstrap required, opened a new log
I20260809 16:39:22.559974 30818 ts_tablet_manager.cc:1397] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:22.560127 30818 raft_consensus.cc:348] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.560205 30818 raft_consensus.cc:374] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.560241 30818 raft_consensus.cc:729] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fe3b628fda4145be4b29dd4d8466d5, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.560287 30818 consensus_queue.cc:260] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.560374 30818 ts_tablet_manager.cc:1428] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:22.560946 30822 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Bootstrap starting.
I20260809 16:39:22.561241 30848 tablet_replica.cc:330] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: stopping tablet replica
I20260809 16:39:22.561365 30848 raft_consensus.cc:2229] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:22.561396 30822 tablet_bootstrap.cc:654] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.561474 30848 raft_consensus.cc:2258] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:22.561996 30822 tablet_bootstrap.cc:492] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: No bootstrap required, opened a new log
I20260809 16:39:22.561982 30848 ts_tablet_manager.cc:1905] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.562059 30822 ts_tablet_manager.cc:1397] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:22.562227 30822 raft_consensus.cc:348] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.562309 30822 raft_consensus.cc:374] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.562338 30822 raft_consensus.cc:729] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44627453b2b949ffa978c541bddb2edc, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.562378 30822 consensus_queue.cc:260] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.562467 30822 ts_tablet_manager.cc:1428] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:22.563999 30848 ts_tablet_manager.cc:1918] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:22.564078 30848 log.cc:1196] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/9951941a2ce1439797dd50dfcea93f4b
I20260809 16:39:22.564370 30848 ts_tablet_manager.cc:1939] T 9951941a2ce1439797dd50dfcea93f4b P 44a13e281ccb4c3caf3347672793b015: Deleting consensus metadata
I20260809 16:39:22.564857 30207 catalog_manager.cc:4841] TS 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499): tablet 9951941a2ce1439797dd50dfcea93f4b (table test-table [id=d1c9203f7c62453a8e0c51f7361815a3]) successfully deleted
I20260809 16:39:22.576056 30824 raft_consensus.cc:482] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:22.576157 30824 raft_consensus.cc:504] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.576514 30824 leader_election.cc:288] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025), 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819)
I20260809 16:39:22.580382 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.576895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:55809 (local address 127.28.62.129:36025)
0809 16:39:22.577001 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.577004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.577015 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:22.577172 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.577173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.577185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.577223 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.577458 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.578087 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.579396 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.579998 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.580093 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.580103 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.580260 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.580304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 16:39:22.580401 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.576681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36025 (local address 127.28.62.130:55809)
0809 16:39:22.576963 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.576974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.577029 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.577293 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.577294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.577296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.577377 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.577378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.578276 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.578278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.579301 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.579304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.580090 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.580092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.580174 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.580179 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.580243 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.580294 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":161,"thread_start_us":72,"threads_started":1}
I20260809 16:39:22.580776 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.576679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36819 (local address 127.28.62.130:52257)
0809 16:39:22.576731 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.576744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.576790 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.577151 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.577152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.577154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.577234 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.577235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.578181 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.578182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.579475 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.579479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.580512 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.580513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.580602 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.580605 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.580666 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.580703 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 16:39:22.580802 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.576970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:52257 (local address 127.28.62.131:36819)
0809 16:39:22.577015 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.577017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.577026 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:22.577055 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.577055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.577064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.577135 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.577313 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.578077 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.579576 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.580183 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.580275 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.580283 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.580688 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.580737 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:22.581027 30727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd644567f6a441dea18e22d53c3df82f" candidate_uuid: "44a13e281ccb4c3caf3347672793b015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44627453b2b949ffa978c541bddb2edc" is_pre_election: true
I20260809 16:39:22.581117 30727 raft_consensus.cc:2454] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a13e281ccb4c3caf3347672793b015 in term 0.
I20260809 16:39:22.581069 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd644567f6a441dea18e22d53c3df82f" candidate_uuid: "44a13e281ccb4c3caf3347672793b015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" is_pre_election: true
I20260809 16:39:22.581246 30406 raft_consensus.cc:2454] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a13e281ccb4c3caf3347672793b015 in term 0.
I20260809 16:39:22.581305 30496 leader_election.cc:302] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44627453b2b949ffa978c541bddb2edc, 44a13e281ccb4c3caf3347672793b015; no voters:
I20260809 16:39:22.581413 30824 raft_consensus.cc:2781] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:22.581446 30824 raft_consensus.cc:482] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:22.581466 30824 raft_consensus.cc:3037] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.582309 30824 raft_consensus.cc:504] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.582445 30824 leader_election.cc:288] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [CANDIDATE]: Term 1 election: Requested vote from peers c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025), 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819)
I20260809 16:39:22.582682 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd644567f6a441dea18e22d53c3df82f" candidate_uuid: "44a13e281ccb4c3caf3347672793b015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fe3b628fda4145be4b29dd4d8466d5"
I20260809 16:39:22.582686 30727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd644567f6a441dea18e22d53c3df82f" candidate_uuid: "44a13e281ccb4c3caf3347672793b015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44627453b2b949ffa978c541bddb2edc"
I20260809 16:39:22.582759 30406 raft_consensus.cc:3037] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.582782 30727 raft_consensus.cc:3037] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.583596 30406 raft_consensus.cc:2454] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a13e281ccb4c3caf3347672793b015 in term 1.
I20260809 16:39:22.583789 30727 raft_consensus.cc:2454] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a13e281ccb4c3caf3347672793b015 in term 1.
I20260809 16:39:22.583801 30497 leader_election.cc:302] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a13e281ccb4c3caf3347672793b015, c7fe3b628fda4145be4b29dd4d8466d5; no voters:
I20260809 16:39:22.583922 30824 raft_consensus.cc:2781] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:22.584041 30824 raft_consensus.cc:686] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 1 LEADER]: Becoming Leader. State: Replica: 44a13e281ccb4c3caf3347672793b015, State: Running, Role: LEADER
I20260809 16:39:22.584107 30824 consensus_queue.cc:237] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.584836 30227 catalog_manager.cc:5495] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44a13e281ccb4c3caf3347672793b015 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "44a13e281ccb4c3caf3347672793b015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } health_report { overall_health: UNKNOWN } } }
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:22.620379 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.611823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37773 (local address 127.0.0.1:35236)
0809 16:39:22.612015 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.612028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.612119 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.612402 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.612403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.612817 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:22.612908 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.612911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.613667 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.613668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.614511 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.614514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.614884 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.614885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.614944 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.615551 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:22.615557 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:22.617371 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:22.619189 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.619197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.619199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.619468 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.619672 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:22.619673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:22.619673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:22.619746 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:22.619995 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:22.620003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:22.620058 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.620143 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.620265 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260809 16:39:22.620465 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.611879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.28.62.190:37773)
0809 16:39:22.611950 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.611953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.612282 (+ 329us) server_negotiation.cc:407] Connection header received
0809 16:39:22.612312 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.612313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.612323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.612384 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.612990 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.613590 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.614612 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.614805 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.614863 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.617480 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.617482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.617482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.617490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.619090 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.619552 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.619552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.619553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.619605 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.619816 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.619816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.619817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.619930 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.619977 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.620267 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.620341 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 16:39:22.622754 30227 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:35236:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 16:39:22.623047 30227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:22.624585 30715 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6ccd1c0324c8e81d1876815f71885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3cc9aebf937491a9d0de71224642b2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:22.624763 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6ccd1c0324c8e81d1876815f71885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.626202 30822 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: Bootstrap starting.
I20260809 16:39:22.626328 30552 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6ccd1c0324c8e81d1876815f71885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3cc9aebf937491a9d0de71224642b2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:22.626490 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6ccd1c0324c8e81d1876815f71885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.626672 30822 tablet_bootstrap.cc:654] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.627413 30822 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: No bootstrap required, opened a new log
I20260809 16:39:22.627470 30822 ts_tablet_manager.cc:1397] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:22.627640 30822 raft_consensus.cc:348] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.627743 30822 raft_consensus.cc:374] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.627766 30822 raft_consensus.cc:729] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44627453b2b949ffa978c541bddb2edc, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.627825 30822 consensus_queue.cc:260] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.627938 30822 ts_tablet_manager.cc:1428] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:22.628149 30819 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: Bootstrap starting.
I20260809 16:39:22.628116 30828 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.628209 30828 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.628229 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 25a6ccd1c0324c8e81d1876815f71885 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3cc9aebf937491a9d0de71224642b2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:22.628357 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a6ccd1c0324c8e81d1876815f71885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:22.628605 30819 tablet_bootstrap.cc:654] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.629586 30818 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: Bootstrap starting.
I20260809 16:39:22.629612 30819 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: No bootstrap required, opened a new log
I20260809 16:39:22.629657 30819 ts_tablet_manager.cc:1397] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260809 16:39:22.629792 30819 raft_consensus.cc:348] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.629851 30819 raft_consensus.cc:374] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.629871 30819 raft_consensus.cc:729] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a13e281ccb4c3caf3347672793b015, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.629913 30819 consensus_queue.cc:260] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
W20260809 16:39:22.629933 30790 tablet.cc:2369] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:22.630005 30819 ts_tablet_manager.cc:1428] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.630059 30818 tablet_bootstrap.cc:654] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:22.630123 30824 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.630184 30824 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.630899 30818 tablet_bootstrap.cc:492] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: No bootstrap required, opened a new log
I20260809 16:39:22.630957 30818 ts_tablet_manager.cc:1397] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:22.631145 30818 raft_consensus.cc:348] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.631222 30818 raft_consensus.cc:374] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:22.631237 30818 raft_consensus.cc:729] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fe3b628fda4145be4b29dd4d8466d5, State: Initialized, Role: FOLLOWER
I20260809 16:39:22.631291 30818 consensus_queue.cc:260] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.631385 30818 ts_tablet_manager.cc:1428] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:22.631403 30830 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.631484 30830 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260809 16:39:22.802270 30462 tablet.cc:2369] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:22.824837 30828 raft_consensus.cc:482] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:22.824939 30828 raft_consensus.cc:504] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.825364 30828 leader_election.cc:288] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025), 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499)
I20260809 16:39:22.828244 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.825667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:33055 (local address 127.28.62.130:40499)
0809 16:39:22.825750 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.825753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.825764 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:22.825828 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.825829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.825843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.825901 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.826057 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.826545 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.827531 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.827874 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.827966 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.827975 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.828129 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.828169 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 16:39:22.828261 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.825574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40499 (local address 127.28.62.131:33055)
0809 16:39:22.825676 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.825688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.825759 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.825905 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.825905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.825907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.825980 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.825982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.826637 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.826638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.827433 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.827436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.827966 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.827967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.828055 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.828058 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.828122 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.828169 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":3}
I20260809 16:39:22.828529 30573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6ccd1c0324c8e81d1876815f71885" candidate_uuid: "44627453b2b949ffa978c541bddb2edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a13e281ccb4c3caf3347672793b015" is_pre_election: true
I20260809 16:39:22.828621 30573 raft_consensus.cc:2454] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44627453b2b949ffa978c541bddb2edc in term 0.
I20260809 16:39:22.828821 30657 leader_election.cc:302] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44627453b2b949ffa978c541bddb2edc, 44a13e281ccb4c3caf3347672793b015; no voters:
I20260809 16:39:22.828948 30828 raft_consensus.cc:2781] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:22.828982 30828 raft_consensus.cc:482] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:22.829021 30828 raft_consensus.cc:3037] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.829206 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.825652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:37851 (local address 127.28.62.129:36025)
0809 16:39:22.825754 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.825756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.826011 (+ 255us) server_negotiation.cc:407] Connection header received
0809 16:39:22.826113 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.826116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.826125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.826191 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:22.826349 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.826823 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.828277 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.828854 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.828940 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.828948 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.829113 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.829153 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 16:39:22.829249 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.825580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36025 (local address 127.28.62.131:37851)
0809 16:39:22.825939 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.825948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.826029 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.826194 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.826194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.826196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:22.826281 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.826282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.826911 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.826912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.828175 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.828178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.828949 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.828950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.829032 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.829037 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.829102 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.829150 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":136,"thread_start_us":154,"threads_started":1}
I20260809 16:39:22.829588 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6ccd1c0324c8e81d1876815f71885" candidate_uuid: "44627453b2b949ffa978c541bddb2edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" is_pre_election: true
I20260809 16:39:22.829702 30406 raft_consensus.cc:2454] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44627453b2b949ffa978c541bddb2edc in term 0.
I20260809 16:39:22.829916 30828 raft_consensus.cc:504] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.830056 30828 leader_election.cc:288] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [CANDIDATE]: Term 1 election: Requested vote from peers c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025), 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499)
I20260809 16:39:22.830204 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6ccd1c0324c8e81d1876815f71885" candidate_uuid: "44627453b2b949ffa978c541bddb2edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fe3b628fda4145be4b29dd4d8466d5"
I20260809 16:39:22.830206 30573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a6ccd1c0324c8e81d1876815f71885" candidate_uuid: "44627453b2b949ffa978c541bddb2edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a13e281ccb4c3caf3347672793b015"
I20260809 16:39:22.830281 30406 raft_consensus.cc:3037] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.830281 30573 raft_consensus.cc:3037] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:22.831039 30406 raft_consensus.cc:2454] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44627453b2b949ffa978c541bddb2edc in term 1.
I20260809 16:39:22.831039 30573 raft_consensus.cc:2454] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44627453b2b949ffa978c541bddb2edc in term 1.
I20260809 16:39:22.831220 30657 leader_election.cc:302] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44627453b2b949ffa978c541bddb2edc, 44a13e281ccb4c3caf3347672793b015; no voters:
I20260809 16:39:22.831316 30828 raft_consensus.cc:2781] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:22.831511 30828 raft_consensus.cc:686] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [term 1 LEADER]: Becoming Leader. State: Replica: 44627453b2b949ffa978c541bddb2edc, State: Running, Role: LEADER
I20260809 16:39:22.831610 30828 consensus_queue.cc:237] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } }
I20260809 16:39:22.831957 30868 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: TxnStatusTablet state changed. Reason: New leader 44627453b2b949ffa978c541bddb2edc. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.832077 30868 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:22.832408 30227 catalog_manager.cc:5495] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc reported cstate change: term changed from 0 to 1, leader changed from <none> to 44627453b2b949ffa978c541bddb2edc (127.28.62.131). New cstate: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260809 16:39:22.834292 30870 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 16:39:22.834416 30870 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 16:39:22.858083 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.855206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.28.62.131:36819)
0809 16:39:22.855267 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.855269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.855301 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:39:22.855395 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.855396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.855417 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.855488 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:22.855612 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.856412 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.857186 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.857496 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.857571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.857746 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.857869 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.857943 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.857982 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.858002 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 16:39:22.858079 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.855140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36819 (local address 127.0.0.1:45526)
0809 16:39:22.855236 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.855266 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.855321 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.855508 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.855509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.855510 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:22.855557 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.855560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.856489 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.856489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.857111 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.857114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.857570 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.857571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.857637 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.857652 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.857935 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.857942 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.857976 (+ 34us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.858018 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 16:39:22.859349 30573 raft_consensus.cc:1261] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Refusing update from remote peer 44627453b2b949ffa978c541bddb2edc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.859366 30406 raft_consensus.cc:1261] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Refusing update from remote peer 44627453b2b949ffa978c541bddb2edc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.859663 30868 consensus_queue.cc:1035] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.859786 30828 consensus_queue.cc:1035] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.860285 30824 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: TxnStatusTablet state changed. Reason: New leader 44627453b2b949ffa978c541bddb2edc. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.860380 30824 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.860433 30830 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: TxnStatusTablet state changed. Reason: New leader 44627453b2b949ffa978c541bddb2edc. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.860520 30830 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.861018 30828 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.861099 30828 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:22.861521 30824 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.861608 30824 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44a13e281ccb4c3caf3347672793b015: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.861845 30868 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.861923 30868 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P 44627453b2b949ffa978c541bddb2edc: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:22.862154 30830 tablet_replica.cc:439] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44627453b2b949ffa978c541bddb2edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 } } peers { permanent_uuid: "44a13e281ccb4c3caf3347672793b015" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40499 } } peers { permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 } } }
I20260809 16:39:22.862236 30830 tablet_replica.cc:442] T 25a6ccd1c0324c8e81d1876815f71885 P c7fe3b628fda4145be4b29dd4d8466d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:22.866245 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.863966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.28.62.130:40499)
0809 16:39:22.864012 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.864015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.864023 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:22.864092 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.864093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.864112 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.864199 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:22.864297 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.864756 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.865470 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.865678 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.865760 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.865928 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.866027 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.866106 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.866138 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.866156 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:22.871083 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.868705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60981 (local address 127.28.62.131:36819)
0809 16:39:22.868749 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.868751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.868924 (+ 173us) server_negotiation.cc:407] Connection header received
0809 16:39:22.868989 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.868990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.869000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.869056 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:22.869187 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.869641 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.870378 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.870567 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.870640 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.870819 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.870853 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.870910 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.870972 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.870992 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:22.871112 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.868595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36819 (local address 127.28.62.130:60981)
0809 16:39:22.868858 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.868867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.868937 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.869059 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.869060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.869062 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:22.869117 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.869118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.869719 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.869720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.870294 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.870296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.870643 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.870645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.870709 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.870728 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.870912 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.870914 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.870967 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.871032 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":180,"threads_started":1}
I20260809 16:39:22.873617 30406 raft_consensus.cc:1261] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Refusing update from remote peer 44a13e281ccb4c3caf3347672793b015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.873624 30727 raft_consensus.cc:1261] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Refusing update from remote peer 44a13e281ccb4c3caf3347672793b015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:22.873883 30853 consensus_queue.cc:1035] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fe3b628fda4145be4b29dd4d8466d5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.874015 30824 consensus_queue.cc:1035] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44627453b2b949ffa978c541bddb2edc" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:22.882582 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.880203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:42549 (local address 127.28.62.130:40499)
0809 16:39:22.880387 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.880407 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.880415 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:22.880436 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.880436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.880446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.880491 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:22.880642 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.881100 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.881811 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.882083 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.882147 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.882356 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.882388 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.882430 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.882491 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.882510 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260809 16:39:22.882587 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.880179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40499 (local address 127.28.62.131:42549)
0809 16:39:22.880259 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:22.880272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:22.880314 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:22.880508 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:22.880510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:22.880511 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:22.880570 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.880571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.881186 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.881187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.881733 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:22.881734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.882179 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.882180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:22.882250 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.882267 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:22.882441 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:22.882442 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:22.882479 (+ 37us) client_negotiation.cc:240] Negotiation successful
0809 16:39:22.882521 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 16:39:22.892483 30227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:22.892616 30227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:22.893363 30227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f48891ef8d7b42b7b8102a2359e861eb: Sending DeleteTablet for 3 replicas of tablet cd644567f6a441dea18e22d53c3df82f
I20260809 16:39:22.893718 30552 tablet_service.cc:1514] Processing DeleteTablet for tablet cd644567f6a441dea18e22d53c3df82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:55864
I20260809 16:39:22.893800 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet cd644567f6a441dea18e22d53c3df82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:46660
I20260809 16:39:22.893880 30848 tablet_replica.cc:330] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: stopping tablet replica
I20260809 16:39:22.893916 30846 tablet_replica.cc:330] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: stopping tablet replica
I20260809 16:39:22.893942 30848 raft_consensus.cc:2229] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:22.893980 30846 raft_consensus.cc:2229] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:22.894027 30846 raft_consensus.cc:2258] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:22.894032 30848 raft_consensus.cc:2258] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:22.894483 30846 ts_tablet_manager.cc:1905] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.894887 30715 tablet_service.cc:1514] Processing DeleteTablet for tablet cd644567f6a441dea18e22d53c3df82f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:22 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:45520
I20260809 16:39:22.895047 30845 tablet_replica.cc:330] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: stopping tablet replica
I20260809 16:39:22.895104 30845 raft_consensus.cc:2229] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:22.895141 30845 raft_consensus.cc:2258] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:22.895469 30845 ts_tablet_manager.cc:1905] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.896308 30848 ts_tablet_manager.cc:1905] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:22.896373 30846 ts_tablet_manager.cc:1918] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:22.896435 30846 log.cc:1196] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/cd644567f6a441dea18e22d53c3df82f
I20260809 16:39:22.896684 30846 ts_tablet_manager.cc:1939] T cd644567f6a441dea18e22d53c3df82f P c7fe3b628fda4145be4b29dd4d8466d5: Deleting consensus metadata
I20260809 16:39:22.897064 30211 catalog_manager.cc:4841] TS c7fe3b628fda4145be4b29dd4d8466d5 (127.28.62.129:36025): tablet cd644567f6a441dea18e22d53c3df82f (table test-table [id=3edb623f25464e2ca9de96ac8ada00ef]) successfully deleted
I20260809 16:39:22.897241 30845 ts_tablet_manager.cc:1918] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:22.897300 30845 log.cc:1196] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/cd644567f6a441dea18e22d53c3df82f
I20260809 16:39:22.897511 30845 ts_tablet_manager.cc:1939] T cd644567f6a441dea18e22d53c3df82f P 44627453b2b949ffa978c541bddb2edc: Deleting consensus metadata
I20260809 16:39:22.897792 30848 ts_tablet_manager.cc:1918] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:22.897850 30848 log.cc:1196] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/cd644567f6a441dea18e22d53c3df82f
I20260809 16:39:22.897857 30210 catalog_manager.cc:4841] TS 44627453b2b949ffa978c541bddb2edc (127.28.62.131:36819): tablet cd644567f6a441dea18e22d53c3df82f (table test-table [id=3edb623f25464e2ca9de96ac8ada00ef]) successfully deleted
I20260809 16:39:22.898053 30848 ts_tablet_manager.cc:1939] T cd644567f6a441dea18e22d53c3df82f P 44a13e281ccb4c3caf3347672793b015: Deleting consensus metadata
I20260809 16:39:22.898514 30207 catalog_manager.cc:4841] TS 44a13e281ccb4c3caf3347672793b015 (127.28.62.130:40499): tablet cd644567f6a441dea18e22d53c3df82f (table test-table [id=3edb623f25464e2ca9de96ac8ada00ef]) successfully deleted
Aug 09 16:39:22 dist-test-slave-wl2h krb5kdc[30176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:22.911145 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.899318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.28.62.129:36025)
0809 16:39:22.899372 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.899374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.903163 (+ 3789us) server_negotiation.cc:407] Connection header received
0809 16:39:22.903249 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.903250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.903263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.903332 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.903959 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.904476 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.905185 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.905393 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.905466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.908312 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.908324 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.908325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.908332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.909905 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.910274 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.910277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.910278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.910338 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.910533 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.910534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.910534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.910678 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.910824 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.910889 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.910970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260809 16:39:22.911384 30445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46680
I20260809 16:39:22.918723 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:22.912288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.28.62.190:37773)
0809 16:39:22.912335 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:22.912338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:22.912513 (+ 175us) server_negotiation.cc:407] Connection header received
0809 16:39:22.912576 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:22.912577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:22.912590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:22.912640 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:22.913226 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.913781 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.914536 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:22.914733 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:22.914801 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:22.915969 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:22.915970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:22.915970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:22.915977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:22.917585 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.917917 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.917918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.917919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.917977 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:22.918204 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:22.918205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:22.918206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:22.918316 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:22.918424 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:22.918526 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:39:22.918607 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260809 16:39:22.918927 30227 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250
I20260809 16:39:22.919536 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 30311
I20260809 16:39:22.940737 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 30475
I20260809 16:39:22.947883 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 30639
I20260809 16:39:22.956032 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 30191
2026-08-09T16:39:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2508 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:23 dist-test-slave-wl2h krb5kdc[30892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:23 dist-test-slave-wl2h krb5kdc[30892](info): set up 2 sockets
Aug 09 16:39:23 dist-test-slave-wl2h krb5kdc[30892](info): commencing operation
krb5kdc: starting...
W20260809 16:39:24.090864 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:24.223701 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40287
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:40287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:24.305512 30905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:24.305734 30905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:24.305758 30905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:24.306968 30905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:24.307034 30905 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 16:39:24.307049 30905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:24.307061 30905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:24.307075 30905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:24.307125 30905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:24.308641 30905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:40287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:24.308847 30905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:24.309024 30905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:24.313966 30910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.313966 30911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.313972 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:24.314044 30905 server_base.cc:1034] running on GCE node
I20260809 16:39:24.314397 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:24.314646 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:24.315798 30905 hybrid_clock.cc:648] HybridClock initialized: now 1786293564315779 us; error 39 us; skew 500 ppm
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:24.318440 30905 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:24.318928 30905 webserver.cc:460] Webserver started at http://127.28.62.190:41523/ using document root <none> and password file <none>
I20260809 16:39:24.319202 30905 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:24.319250 30905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:24.319404 30905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:24.320499 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "c30d4b390f2c4ad5af64faa94aa84b2a"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "10123970cc7d5d9a95ca2839e8b5b49b"
server_key_iv: "855966269723e2aea54bf57048a28d0f"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.320885 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "c30d4b390f2c4ad5af64faa94aa84b2a"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "10123970cc7d5d9a95ca2839e8b5b49b"
server_key_iv: "855966269723e2aea54bf57048a28d0f"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.322237 30905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:24.322978 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.323179 30905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:24.323284 30905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "c30d4b390f2c4ad5af64faa94aa84b2a"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "10123970cc7d5d9a95ca2839e8b5b49b"
server_key_iv: "855966269723e2aea54bf57048a28d0f"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.323357 30905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:24.349270 30905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:24.351719 30905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:24.351890 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:24.356774 30905 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:40287
I20260809 16:39:24.356838 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:40287 every 8 connection(s)
I20260809 16:39:24.357151 30905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:24.357746 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:24.359248 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 30905
I20260809 16:39:24.359349 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:24.359520 28922 external_mini_cluster.cc:1428] Setting key 3a38135ae65777b0bfe00213c29f9eb1
I20260809 16:39:24.360531 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Bootstrap starting.
I20260809 16:39:24.361243 30996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:24.361558 30996 log.cc:824] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:24.362278 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: No bootstrap required, opened a new log
I20260809 16:39:24.364171 30996 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } }
I20260809 16:39:24.364334 30996 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:24.364382 30996 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30d4b390f2c4ad5af64faa94aa84b2a, State: Initialized, Role: FOLLOWER
I20260809 16:39:24.364529 30996 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } }
I20260809 16:39:24.364614 30996 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:24.364660 30996 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:24.364727 30996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:24.365273 30996 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } }
I20260809 16:39:24.365388 30996 leader_election.cc:302] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30d4b390f2c4ad5af64faa94aa84b2a; no voters:
I20260809 16:39:24.365564 30996 leader_election.cc:288] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:24.365605 31002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:24.365787 31002 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [term 1 LEADER]: Becoming Leader. State: Replica: c30d4b390f2c4ad5af64faa94aa84b2a, State: Running, Role: LEADER
I20260809 16:39:24.365839 30996 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:24.365875 31002 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } }
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:24.366245 31003 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } } }
I20260809 16:39:24.366319 31003 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:24.366547 31004 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30d4b390f2c4ad5af64faa94aa84b2a. Latest consensus state: current_term: 1 leader_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30d4b390f2c4ad5af64faa94aa84b2a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40287 } } }
I20260809 16:39:24.366688 31004 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:24.367063 31011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:24.368046 31011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:24.369666 31011 catalog_manager.cc:1347] Generated new cluster ID: e55495ae20a74adbb9a69bc68e876cf9
I20260809 16:39:24.369721 31011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:24.370584 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.360067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.28.62.190:40287)
0809 16:39:24.360299 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.360307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.360332 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:39:24.360794 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.360801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.360848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.360991 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.361684 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.362677 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.363474 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.363893 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.364013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.366672 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.366677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.366677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.366689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.368870 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.369264 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.369267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.369268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.369378 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.369639 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.369640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.369641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.369945 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.370103 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.370250 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.370394 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260809 16:39:24.384619 31011 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:24.385416 31011 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:24.389588 31011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Generated new TSK 0
I20260809 16:39:24.389768 31011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:24.426211 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40287
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:24.509042 31027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:24.509250 31027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:24.509276 31027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:24.510416 31027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:24.510458 31027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:24.510479 31027 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 16:39:24.510491 31027 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:24.510535 31027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:24.512079 31027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:24.512308 31027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:24.512549 31027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:24.517597 31036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.517618 31034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.517597 31032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:24.517836 31027 server_base.cc:1034] running on GCE node
I20260809 16:39:24.518054 31027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:24.518294 31027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:24.519455 31027 hybrid_clock.cc:648] HybridClock initialized: now 1786293564519446 us; error 35 us; skew 500 ppm
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:24.521974 31027 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:24.522447 31027 webserver.cc:460] Webserver started at http://127.28.62.129:34795/ using document root <none> and password file <none>
I20260809 16:39:24.522675 31027 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:24.522742 31027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:24.522893 31027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:24.524117 31027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "a87b6747a88c86d568c23f2f58f436bc"
server_key_iv: "58a7a6289457d7c7276ecff474b32dbc"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.524493 31027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "a87b6747a88c86d568c23f2f58f436bc"
server_key_iv: "58a7a6289457d7c7276ecff474b32dbc"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.525949 31027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:24.526782 31045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.527159 31027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:24.527261 31027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "a87b6747a88c86d568c23f2f58f436bc"
server_key_iv: "58a7a6289457d7c7276ecff474b32dbc"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.527345 31027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:24.540081 31027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:24.542598 31027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:24.542773 31027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:24.543411 31027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:24.543455 31027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.543481 31027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:24.543524 31027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.551069 31027 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:39085
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:24.551179 31176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:39085 every 8 connection(s)
I20260809 16:39:24.551532 31027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:24.552085 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31027
I20260809 16:39:24.552201 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:24.552452 28922 external_mini_cluster.cc:1428] Setting key 82514d6d82a6acff42e8150572de1c96
I20260809 16:39:24.555322 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.543930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:38105 (local address 127.28.62.190:40287)
0809 16:39:24.544005 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.544008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.544176 (+ 168us) server_negotiation.cc:407] Connection header received
0809 16:39:24.544673 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.544675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.544691 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.544779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.545615 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.546567 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.548200 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.548629 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.548731 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.551811 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.551812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.551813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.551821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.553747 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.554210 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.554211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.554211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.554282 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.554558 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.554559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.554560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.554690 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.554774 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.555117 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.555221 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:39:24.555516 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.543882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.129:38105)
0809 16:39:24.544045 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.544086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.544579 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.544839 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.544842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.545168 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.545498 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.545505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.546710 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.546712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.548096 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.548099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.548775 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.548776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.548934 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.549532 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.549542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.551697 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.553853 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.553862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.553864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.554114 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.554352 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.554353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.554354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.554475 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.554763 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.554771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.554922 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.555091 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.555265 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260809 16:39:24.556236 31178 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40287
I20260809 16:39:24.556329 31178 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:24.556514 31178 heartbeater.cc:511] Master 127.28.62.190:40287 requested a full tablet report, sending...
I20260809 16:39:24.557058 30937 ts_manager.cc:194] Registered new tserver with Master: 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085)
I20260809 16:39:24.557868 30937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:38105
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:39:24.563699 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.557171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.129:60897)
0809 16:39:24.557348 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.557360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.557424 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.557548 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.557549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.557765 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.557833 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.557835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.558695 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.558696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.559661 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.559665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.560017 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.560018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.560107 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.560569 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.560575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.560971 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.562614 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.562616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.562616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.562837 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.563061 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.563062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.563062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.563133 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.563427 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.563428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.563479 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.563553 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.563632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260809 16:39:24.563721 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.557241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:60897 (local address 127.28.62.190:40287)
0809 16:39:24.557282 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.557284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.557430 (+ 146us) server_negotiation.cc:407] Connection header received
0809 16:39:24.557478 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.557479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.557489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.557543 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.557912 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.558569 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.559748 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.559934 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.559999 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.561075 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.561076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.561076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.561084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.562529 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.562918 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.562919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.562919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.562969 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.563223 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.563224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.563225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.563357 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.563410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.563584 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.563648 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:24.582683 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40287
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:24.666281 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:24.666491 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:24.666515 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:24.667943 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:24.667994 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:24.668015 31187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 16:39:24.668027 31187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:24.668045 31187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:24.669469 31187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:24.669662 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:24.669852 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:24.674906 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.674939 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:24.674973 31187 server_base.cc:1034] running on GCE node
W20260809 16:39:24.675001 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:24.675429 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:24.675700 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:24.676860 31187 hybrid_clock.cc:648] HybridClock initialized: now 1786293564676855 us; error 37 us; skew 500 ppm
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:24.679632 31187 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:24.680135 31187 webserver.cc:460] Webserver started at http://127.28.62.130:37733/ using document root <none> and password file <none>
I20260809 16:39:24.680387 31187 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:24.680456 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:24.680598 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:24.681680 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "22a6158b551a4ea9823106a59860f5d2"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "e6c9f9cf300d0d5db3f9c3ac8589fdd3"
server_key_iv: "8a053bf877a6ab632ed2fa1f9987aff4"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.682061 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "22a6158b551a4ea9823106a59860f5d2"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "e6c9f9cf300d0d5db3f9c3ac8589fdd3"
server_key_iv: "8a053bf877a6ab632ed2fa1f9987aff4"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.683558 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:24.684304 31203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.684473 31187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:24.684561 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "22a6158b551a4ea9823106a59860f5d2"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "e6c9f9cf300d0d5db3f9c3ac8589fdd3"
server_key_iv: "8a053bf877a6ab632ed2fa1f9987aff4"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.684636 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:24.696043 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:24.697978 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:24.698145 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:24.698699 31187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:24.698763 31187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.698805 31187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:24.698835 31187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:24.706588 31187 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:35551
I20260809 16:39:24.706678 31331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:35551 every 8 connection(s)
I20260809 16:39:24.707069 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:24.708119 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31187
I20260809 16:39:24.708225 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:24.708436 28922 external_mini_cluster.cc:1428] Setting key cce3d3e51a27277799d3e986afa3d7f9
I20260809 16:39:24.709440 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.699152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:39101 (local address 127.28.62.190:40287)
0809 16:39:24.699222 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.699224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.699494 (+ 270us) server_negotiation.cc:407] Connection header received
0809 16:39:24.700033 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.700035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.700048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.700096 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.700738 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.701270 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.702339 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.702546 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.702622 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.706073 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.706074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.706074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.706081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.707806 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.708331 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.708333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.708333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.708399 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.708786 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.708787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.708787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.708899 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.708989 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.709264 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.709347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:24.709617 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.699143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.130:39101)
0809 16:39:24.699356 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.699396 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.699928 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.700164 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.700167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.700459 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.700637 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.700642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.701370 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.701371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.702232 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.702234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.702631 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.702632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.702760 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.703335 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.703347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.705981 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.707878 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.707882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.707884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.708215 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.708489 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.708490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.708490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.708632 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.708981 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.708988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.709124 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.709235 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.709394 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
I20260809 16:39:24.711192 31332 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40287
I20260809 16:39:24.711282 31332 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:24.711474 31332 heartbeater.cc:511] Master 127.28.62.190:40287 requested a full tablet report, sending...
I20260809 16:39:24.712319 30937 ts_manager.cc:194] Registered new tserver with Master: 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551)
I20260809 16:39:24.712791 30937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:39101
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:39:24.718173 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.712304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:58325 (local address 127.28.62.190:40287)
0809 16:39:24.712344 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.712345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.712433 (+ 88us) server_negotiation.cc:407] Connection header received
0809 16:39:24.712541 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.712542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.712553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.712622 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.713029 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.713525 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.714422 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.714623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.714695 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.715651 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.715653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.715653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.715660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.716988 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.717360 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.717361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.717361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.717420 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.717617 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.717618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.717618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.717763 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.717835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.718000 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.718084 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:24.718195 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.712214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.130:58325)
0809 16:39:24.712344 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.712355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.712458 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.712626 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.712627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.712887 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.712954 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.712956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.713631 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.713632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.714335 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.714337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.714694 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.714695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.714778 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.715255 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.715259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.715529 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.717097 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.717098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.717099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.717275 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.717483 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.717484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.717485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.717547 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.717840 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.717841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.717886 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.717960 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.718104 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:24.738076 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40287
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:24.820705 31341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:24.820905 31341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:24.820930 31341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:24.821985 31341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:24.822018 31341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:24.822038 31341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 16:39:24.822052 31341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:24.822067 31341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:24.823626 31341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:24.823810 31341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:24.823987 31341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:24.829013 31347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.829025 31349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:24.829047 31346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:24.829352 31341 server_base.cc:1034] running on GCE node
I20260809 16:39:24.829568 31341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:24.829829 31341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:24.830986 31341 hybrid_clock.cc:648] HybridClock initialized: now 1786293564830987 us; error 42 us; skew 500 ppm
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:24.833477 31341 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:24.833899 31341 webserver.cc:460] Webserver started at http://127.28.62.131:34717/ using document root <none> and password file <none>
I20260809 16:39:24.834126 31341 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:24.834194 31341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:24.834347 31341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:24.835457 31341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "95950a97b93147f8b03dc76c12a814c0"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "72a72b326b28fc0285e36a237d6ab85c"
server_key_iv: "43f2c89fb02035f669f26eb96bbdc18d"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.835811 31341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "95950a97b93147f8b03dc76c12a814c0"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "72a72b326b28fc0285e36a237d6ab85c"
server_key_iv: "43f2c89fb02035f669f26eb96bbdc18d"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.837316 31341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260809 16:39:24.838022 31357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.838192 31341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.838302 31341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "95950a97b93147f8b03dc76c12a814c0"
format_stamp: "Formatted at 2026-08-09 16:39:24 on dist-test-slave-wl2h"
server_key: "72a72b326b28fc0285e36a237d6ab85c"
server_key_iv: "43f2c89fb02035f669f26eb96bbdc18d"
server_key_version: "encryptionkey@0"
I20260809 16:39:24.838375 31341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:24.846939 31341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:24.848824 31341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:24.848985 31341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:24.849488 31341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:24.849555 31341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:24.849591 31341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:24.849632 31341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:24.857005 31341 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:38457
I20260809 16:39:24.857076 31490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:38457 every 8 connection(s)
I20260809 16:39:24.857431 31341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:24.860124 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.849985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41819 (local address 127.28.62.190:40287)
0809 16:39:24.850063 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.850066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.850237 (+ 171us) server_negotiation.cc:407] Connection header received
0809 16:39:24.850704 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.850705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.850720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.850811 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.851474 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.852336 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.853925 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.854270 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.854358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.857139 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.857140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.857140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.857149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.858749 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.859270 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.859271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.859271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.859331 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.859538 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.859538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.859538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.859650 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.859699 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.859979 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.860047 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:39:24.860306 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.849944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.131:41819)
0809 16:39:24.850100 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.850139 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.850606 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.850857 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.850860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.851164 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.851385 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.851390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.852454 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.852455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.853825 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.853828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.854379 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.854379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.854525 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.855091 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.855100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.857047 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.858844 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.858851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.858853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.859172 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.859391 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.859392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.859392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.859492 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.859730 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.859736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.859852 (+ 116us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.859949 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.860072 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260809 16:39:24.860841 31491 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40287
I20260809 16:39:24.860921 31491 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:24.861101 31491 heartbeater.cc:511] Master 127.28.62.190:40287 requested a full tablet report, sending...
I20260809 16:39:24.861501 30938 ts_manager.cc:194] Registered new tserver with Master: 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457)
I20260809 16:39:24.862095 30938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:41819
I20260809 16:39:24.863369 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31341
I20260809 16:39:24.863461 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:24.863667 28922 external_mini_cluster.cc:1428] Setting key 588d01184102d628afc9400957409276
I20260809 16:39:24.864634 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:24.868886 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.861985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:57225 (local address 127.28.62.190:40287)
0809 16:39:24.862020 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.862022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.862151 (+ 129us) server_negotiation.cc:407] Connection header received
0809 16:39:24.862204 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.862205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.862216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.862277 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.862653 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.863233 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.864077 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.864340 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.864410 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.865272 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.865273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.865273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.865281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.867599 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.868008 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.868010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.868010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.868087 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.868311 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.868312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.868312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.868441 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.868525 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.868754 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.868823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 16:39:24.868995 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.861933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.28.62.131:57225)
0809 16:39:24.862080 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.862092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.862154 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.862284 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.862285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.862517 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:24.862579 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.862581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.863323 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.863323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.864003 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.864005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.864409 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.864410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.864485 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.864910 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:24.864914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:24.865179 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:24.867712 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.867713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.867713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.867910 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.868166 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:24.868167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:24.868167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:24.868238 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:24.868516 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:24.868517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:24.868583 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.868669 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.868821 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 16:39:24 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:24.891357 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.883521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.28.62.190:40287)
0809 16:39:24.883584 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.883586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.883618 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:39:24.883719 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.883720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.883738 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.883805 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:24.884460 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.885009 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.885824 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.886049 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.886131 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.888663 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:24.888665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:24.888665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:24.888677 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:24.890203 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.890544 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.890545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.890546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.890603 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:24.890801 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:24.890802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:24.890803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:24.890920 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:24.891017 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.891167 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.891252 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:24.893055 30938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:24.894698 30938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:24.901274 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.0.0.1:50480)
0809 16:39:24.897276 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.897320 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.897386 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.897730 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.897733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.897735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:24.897808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.897810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.899087 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.899088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.900219 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.900222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.900965 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.900966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.901045 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.901068 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.901143 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.901194 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":46,"threads_started":1}
I20260809 16:39:24.901315 31516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.28.62.130:35551)
0809 16:39:24.897558 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.897562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.897570 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:24.897602 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.897603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.897616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.897712 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:24.897936 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.898989 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.900304 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.900880 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.900957 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.901016 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.901167 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.901207 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260809 16:39:24.902098 31256 tablet_service.cc:1467] Processing CreateTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 (DEFAULT_TABLE table=test-table [id=2d35a151427d46e98576b33dad738436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:24.902287 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:39085 (local address 127.0.0.1:37662)
0809 16:39:24.897804 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.897815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.897856 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.898436 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.898437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.898439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:24.898481 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.898483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.899521 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.899522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.900682 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.900684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.901934 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.901935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.902026 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.902031 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.902160 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.902213 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":89,"threads_started":1}
I20260809 16:39:24.902334 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.28.62.129:39085)
0809 16:39:24.898293 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.898297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.898306 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:24.898344 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.898346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.898358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.898439 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:24.898539 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.899433 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.900765 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.901841 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.901913 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.901966 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.902187 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.902230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":76,"threads_started":1}
I20260809 16:39:24.902503 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0379769a06483b950a73a4d5e3d8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:24.903452 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:38457 (local address 127.0.0.1:33194)
0809 16:39:24.897448 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:24.897460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:24.897515 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:24.898159 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:24.898160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:24.898162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:24.898214 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.898216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.899388 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.899389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.900615 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:24.900618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.903233 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.903234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:24.903297 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.903301 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:24.903354 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:24.903391 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":32,"thread_start_us":137,"threads_started":1}
I20260809 16:39:24.903729 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:24.897248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.28.62.131:38457)
0809 16:39:24.898007 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:24.898012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:24.898020 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:24.898058 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:24.898059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:24.898071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:24.898157 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:24.898308 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.899310 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.900719 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:24.901572 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:24.901643 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:24.901699 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:24.903593 (+ 1894us) server_negotiation.cc:299] Negotiation successful
0809 16:39:24.903633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":103,"threads_started":1}
I20260809 16:39:24.903760 31103 tablet_service.cc:1467] Processing CreateTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 (DEFAULT_TABLE table=test-table [id=2d35a151427d46e98576b33dad738436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:24.904115 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0379769a06483b950a73a4d5e3d8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:24.906116 31517 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Bootstrap starting.
I20260809 16:39:24.906708 31517 tablet_bootstrap.cc:654] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:24.907065 31517 log.cc:824] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:24.907888 31517 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: No bootstrap required, opened a new log
I20260809 16:39:24.907970 31517 ts_tablet_manager.cc:1397] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:24.908087 31519 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Bootstrap starting.
I20260809 16:39:24.908722 31519 tablet_bootstrap.cc:654] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:24.908819 31413 tablet_service.cc:1467] Processing CreateTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 (DEFAULT_TABLE table=test-table [id=2d35a151427d46e98576b33dad738436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:24.909112 31519 log.cc:824] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:24.909158 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0379769a06483b950a73a4d5e3d8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:24.909860 31519 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: No bootstrap required, opened a new log
I20260809 16:39:24.909945 31519 ts_tablet_manager.cc:1397] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:24.910444 31517 raft_consensus.cc:348] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.910614 31517 raft_consensus.cc:374] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:24.910655 31517 raft_consensus.cc:729] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Initialized, Role: FOLLOWER
I20260809 16:39:24.910794 31517 consensus_queue.cc:260] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.911132 31517 ts_tablet_manager.cc:1428] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:24.911206 31332 heartbeater.cc:503] Master 127.28.62.190:40287 was elected leader, sending a full tablet report...
I20260809 16:39:24.912015 31519 raft_consensus.cc:348] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.912259 31519 raft_consensus.cc:374] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:24.912350 31519 raft_consensus.cc:729] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:24.912505 31519 consensus_queue.cc:260] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.912822 31519 ts_tablet_manager.cc:1428] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:24.912868 31523 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Bootstrap starting.
I20260809 16:39:24.912962 31178 heartbeater.cc:503] Master 127.28.62.190:40287 was elected leader, sending a full tablet report...
I20260809 16:39:24.913918 31523 tablet_bootstrap.cc:654] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:24.914353 31523 log.cc:824] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:24.915307 31523 tablet_bootstrap.cc:492] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: No bootstrap required, opened a new log
I20260809 16:39:24.915391 31523 ts_tablet_manager.cc:1397] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:39:24.916924 31523 raft_consensus.cc:348] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.917095 31523 raft_consensus.cc:374] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:24.917150 31523 raft_consensus.cc:729] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Initialized, Role: FOLLOWER
I20260809 16:39:24.917280 31523 consensus_queue.cc:260] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:24.917583 31523 ts_tablet_manager.cc:1428] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:24.917655 31491 heartbeater.cc:503] Master 127.28.62.190:40287 was elected leader, sending a full tablet report...
W20260809 16:39:24.957772 31333 tablet.cc:2369] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:39:25.052603 31179 tablet.cc:2369] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:25.072258 31521 raft_consensus.cc:482] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:25.072384 31521 raft_consensus.cc:504] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:25.072839 31521 leader_election.cc:288] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457), 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085)
I20260809 16:39:25.075919 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.073064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:39085 (local address 127.28.62.130:49929)
0809 16:39:25.073121 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.073135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.073190 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.073431 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.073433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.073435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:25.073515 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.073517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.074213 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.074215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.075114 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.075116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.075647 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.075731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.075735 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.075794 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.075822 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":27}
I20260809 16:39:25.075922 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.073240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:49929 (local address 127.28.62.129:39085)
0809 16:39:25.073284 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.073287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.073298 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:25.073331 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.073332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.073344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.073412 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:25.073594 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.074138 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075199 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.075561 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075619 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.075625 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.075805 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.075849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:25.076015 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.073148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:59793 (local address 127.28.62.131:38457)
0809 16:39:25.073208 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.073210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.073419 (+ 209us) server_negotiation.cc:407] Connection header received
0809 16:39:25.073453 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.073454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.073463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.073528 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:25.073741 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.074248 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075248 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.075632 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075704 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.075714 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.075898 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.075944 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 16:39:25.076072 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.073070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:38457 (local address 127.28.62.130:59793)
0809 16:39:25.073331 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.073342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.073404 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.073538 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.073539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.073571 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:25.073657 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.073659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.074329 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.074330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.075119 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.075121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.075723 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.075724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.075812 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.075816 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.075872 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.075910 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":79,"thread_start_us":108,"threads_started":1}
I20260809 16:39:25.076288 31127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0379769a06483b950a73a4d5e3d8f3" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" is_pre_election: true
I20260809 16:39:25.076444 31127 raft_consensus.cc:2454] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 0.
I20260809 16:39:25.076468 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0379769a06483b950a73a4d5e3d8f3" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0" is_pre_election: true
I20260809 16:39:25.076603 31434 raft_consensus.cc:2454] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 0.
I20260809 16:39:25.076685 31207 leader_election.cc:302] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 5ddad79fcb6a4675800ebcb1898d66c6; no voters:
I20260809 16:39:25.076822 31521 raft_consensus.cc:2781] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:25.076877 31521 raft_consensus.cc:482] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:25.076911 31521 raft_consensus.cc:3037] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.077886 31521 raft_consensus.cc:504] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:25.078027 31521 leader_election.cc:288] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 election: Requested vote from peers 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457), 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085)
I20260809 16:39:25.078197 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0379769a06483b950a73a4d5e3d8f3" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0"
I20260809 16:39:25.078341 31434 raft_consensus.cc:3037] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.078361 31127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0379769a06483b950a73a4d5e3d8f3" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
I20260809 16:39:25.078444 31127 raft_consensus.cc:3037] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.079188 31434 raft_consensus.cc:2454] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 1.
I20260809 16:39:25.079367 31127 raft_consensus.cc:2454] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 1.
I20260809 16:39:25.079375 31204 leader_election.cc:302] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:25.079494 31521 raft_consensus.cc:2781] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:25.079661 31521 raft_consensus.cc:686] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Running, Role: LEADER
I20260809 16:39:25.079787 31521 consensus_queue.cc:237] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } }
I20260809 16:39:25.080613 30938 catalog_manager.cc:5495] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } health_report { overall_health: UNKNOWN } } }
W20260809 16:39:25.108062 31492 tablet.cc:2369] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 16:39:25 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:25.159646 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.151782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40287 (local address 127.0.0.1:40986)
0809 16:39:25.151951 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.151959 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.152015 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.152179 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.152180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.152505 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:25.152558 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.152560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.153186 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.153187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.153926 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.153929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.154499 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.154500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.154595 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.155242 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:25.155246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:25.156718 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:25.158243 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:25.158250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:25.158252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:25.158450 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:25.158687 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:25.158688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:25.158688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:25.158875 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:25.159292 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:25.159299 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:25.159344 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.159422 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.159516 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260809 16:39:25.159651 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.151881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.28.62.190:40287)
0809 16:39:25.151947 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.151952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.152002 (+ 50us) server_negotiation.cc:407] Connection header received
0809 16:39:25.152085 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.152086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.152095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.152159 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:25.152634 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.153114 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.154067 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.154415 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.154485 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.156807 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:25.156808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:25.156809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:25.156815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:25.158152 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:25.158524 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:25.158524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:25.158525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:25.158576 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:25.158989 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:25.158990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:25.158990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:25.159200 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:25.159270 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.159476 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.159569 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":37}
I20260809 16:39:25.161893 30938 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:40986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 16:39:25.162191 30938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:25.163897 31103 tablet_service.cc:1467] Processing CreateTablet for tablet 35436d8a727e42c68cc267d0eb63bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e187f9c6e264a9bad2aa456ad9d77a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:25.163924 31256 tablet_service.cc:1467] Processing CreateTablet for tablet 35436d8a727e42c68cc267d0eb63bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e187f9c6e264a9bad2aa456ad9d77a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:25.163923 31413 tablet_service.cc:1467] Processing CreateTablet for tablet 35436d8a727e42c68cc267d0eb63bb08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e187f9c6e264a9bad2aa456ad9d77a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:25.164085 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35436d8a727e42c68cc267d0eb63bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.164085 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35436d8a727e42c68cc267d0eb63bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.164086 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35436d8a727e42c68cc267d0eb63bb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.165335 31519 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: Bootstrap starting.
I20260809 16:39:25.165762 31519 tablet_bootstrap.cc:654] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.165776 31517 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: Bootstrap starting.
I20260809 16:39:25.166239 31517 tablet_bootstrap.cc:654] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.166637 31523 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: Bootstrap starting.
I20260809 16:39:25.166967 31517 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: No bootstrap required, opened a new log
I20260809 16:39:25.167052 31517 ts_tablet_manager.cc:1397] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:25.167276 31523 tablet_bootstrap.cc:654] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.167236 31517 raft_consensus.cc:348] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.167330 31517 raft_consensus.cc:374] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.167354 31517 raft_consensus.cc:729] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.167414 31517 consensus_queue.cc:260] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.167524 31517 ts_tablet_manager.cc:1428] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:25.167584 31521 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.167703 31521 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.167977 31519 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: No bootstrap required, opened a new log
I20260809 16:39:25.168053 31519 ts_tablet_manager.cc:1397] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:25.168216 31519 raft_consensus.cc:348] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.168296 31519 raft_consensus.cc:374] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.168316 31519 raft_consensus.cc:729] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.168391 31519 consensus_queue.cc:260] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.168488 31519 ts_tablet_manager.cc:1428] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:25.168601 31524 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.168720 31524 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.168731 31523 tablet_bootstrap.cc:492] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: No bootstrap required, opened a new log
I20260809 16:39:25.168778 31523 ts_tablet_manager.cc:1397] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:25.168926 31523 raft_consensus.cc:348] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.169003 31523 raft_consensus.cc:374] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.169028 31523 raft_consensus.cc:729] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.169070 31523 consensus_queue.cc:260] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.169180 31523 ts_tablet_manager.cc:1428] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:25.169317 31526 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.169407 31526 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.394084 31521 raft_consensus.cc:482] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:25.394189 31521 raft_consensus.cc:504] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.394372 31521 leader_election.cc:288] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457)
I20260809 16:39:25.394625 31127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35436d8a727e42c68cc267d0eb63bb08" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" is_pre_election: true
I20260809 16:39:25.394634 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35436d8a727e42c68cc267d0eb63bb08" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0" is_pre_election: true
I20260809 16:39:25.394755 31127 raft_consensus.cc:2454] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 0.
I20260809 16:39:25.394754 31434 raft_consensus.cc:2454] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 0.
I20260809 16:39:25.394960 31204 leader_election.cc:302] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:25.395104 31521 raft_consensus.cc:2781] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:25.395159 31521 raft_consensus.cc:482] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:25.395190 31521 raft_consensus.cc:3037] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.395917 31521 raft_consensus.cc:504] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.396070 31521 leader_election.cc:288] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457)
I20260809 16:39:25.396241 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35436d8a727e42c68cc267d0eb63bb08" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0"
I20260809 16:39:25.396327 31434 raft_consensus.cc:3037] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.396286 31127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35436d8a727e42c68cc267d0eb63bb08" candidate_uuid: "22a6158b551a4ea9823106a59860f5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
I20260809 16:39:25.396385 31127 raft_consensus.cc:3037] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.396974 31434 raft_consensus.cc:2454] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 1.
I20260809 16:39:25.397097 31127 raft_consensus.cc:2454] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a6158b551a4ea9823106a59860f5d2 in term 1.
I20260809 16:39:25.397145 31204 leader_election.cc:302] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:25.397233 31521 raft_consensus.cc:2781] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:25.397286 31521 raft_consensus.cc:686] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Running, Role: LEADER
I20260809 16:39:25.397347 31521 consensus_queue.cc:237] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.397650 31529 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: TxnStatusTablet state changed. Reason: New leader 22a6158b551a4ea9823106a59860f5d2. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.397746 31529 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:25.397969 30937 catalog_manager.cc:5495] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:25.398267 31543 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 16:39:25.398401 31543 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 16:39:25.452872 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.450016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.28.62.130:35551)
0809 16:39:25.450102 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.450104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.450113 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:25.450182 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.450182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.450198 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.450264 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.450423 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.450898 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.451983 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.452187 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.452242 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.452497 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.452604 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.452686 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.452782 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.452804 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
I20260809 16:39:25.452909 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.449920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.0.0.1:60070)
0809 16:39:25.450012 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.450027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.450090 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.450292 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.450293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.450294 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:25.450350 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.450352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.450976 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.450977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.451902 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.451905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.452275 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.452276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.452348 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.452365 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.452699 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.452711 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.452773 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.452823 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 16:39:25.454763 31127 raft_consensus.cc:1261] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Refusing update from remote peer 22a6158b551a4ea9823106a59860f5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.454769 31434 raft_consensus.cc:1261] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Refusing update from remote peer 22a6158b551a4ea9823106a59860f5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.455153 31521 consensus_queue.cc:1035] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.455313 31529 consensus_queue.cc:1035] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.456446 31526 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: TxnStatusTablet state changed. Reason: New leader 22a6158b551a4ea9823106a59860f5d2. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.456538 31526 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.456626 31524 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: TxnStatusTablet state changed. Reason: New leader 22a6158b551a4ea9823106a59860f5d2. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.456768 31524 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.458086 31521 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.458170 31521 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:25.458266 31529 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.458389 31529 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 22a6158b551a4ea9823106a59860f5d2: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:25.458686 31526 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.458778 31526 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 95950a97b93147f8b03dc76c12a814c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.458952 31524 tablet_replica.cc:439] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22a6158b551a4ea9823106a59860f5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } } }
I20260809 16:39:25.459128 31547 mvcc.cc:205] Tried to move back new op lower bound from 7316658444099629056 to 7316658443867979776. Current Snapshot: MvccSnapshot[applied={T|T < 7316658444099629056}]
I20260809 16:39:25.459190 31524 tablet_replica.cc:442] T 35436d8a727e42c68cc267d0eb63bb08 P 5ddad79fcb6a4675800ebcb1898d66c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:25.459224 31548 mvcc.cc:205] Tried to move back new op lower bound from 7316658444099629056 to 7316658443867979776. Current Snapshot: MvccSnapshot[applied={T|T < 7316658444099629056}]
I20260809 16:39:25.459725 31549 mvcc.cc:205] Tried to move back new op lower bound from 7316658444099629056 to 7316658443867979776. Current Snapshot: MvccSnapshot[applied={T|T < 7316658444099629056}]
I20260809 16:39:25.464596 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.461512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.28.62.130:35551)
0809 16:39:25.461565 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.461567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.461699 (+ 132us) server_negotiation.cc:407] Connection header received
0809 16:39:25.461759 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.461760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.461769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.461866 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.461991 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.462571 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.463708 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.463961 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.464021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.464290 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.464320 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.464378 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.464496 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.464516 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:25.469426 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.467106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:57507 (local address 127.28.62.130:35551)
0809 16:39:25.467140 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.467142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.467293 (+ 151us) server_negotiation.cc:407] Connection header received
0809 16:39:25.467365 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.467366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.467374 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.467413 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.467552 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.467995 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.468735 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.468945 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.469022 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.469203 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.469232 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.469281 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.469341 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.469357 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260809 16:39:25.469547 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.467073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.28.62.130:57507)
0809 16:39:25.467212 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.467228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.467308 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.467427 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.467428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.467430 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:25.467481 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.467482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.468078 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.468079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.468649 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.468651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.469035 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.469036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.469100 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.469116 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.469280 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.469288 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.469338 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.469379 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260809 16:39:25.471792 31434 raft_consensus.cc:1261] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Refusing update from remote peer 22a6158b551a4ea9823106a59860f5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.471800 31127 raft_consensus.cc:1261] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Refusing update from remote peer 22a6158b551a4ea9823106a59860f5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.472023 31521 consensus_queue.cc:1035] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.472155 31529 consensus_queue.cc:1035] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.485854 30938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:25.485954 30938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:25.486686 30938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Sending DeleteTablet for 3 replicas of tablet ab0379769a06483b950a73a4d5e3d8f3
I20260809 16:39:25.486979 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:25 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50480
I20260809 16:39:25.487078 31413 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:25 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33194
I20260809 16:39:25.487115 31103 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0379769a06483b950a73a4d5e3d8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:25 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37662
I20260809 16:39:25.487382 31570 tablet_replica.cc:330] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: stopping tablet replica
I20260809 16:39:25.487463 31567 tablet_replica.cc:330] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: stopping tablet replica
I20260809 16:39:25.487548 31570 raft_consensus.cc:2229] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:25.487629 31567 raft_consensus.cc:2229] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:25.487694 31570 raft_consensus.cc:2258] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:25.487704 30938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 16:39:25.487805 31567 raft_consensus.cc:2258] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 16:39:25.487874 30938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:25.488211 31567 ts_tablet_manager.cc:1905] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:25.488389 31570 ts_tablet_manager.cc:1905] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:25.489303 31103 tablet_service.cc:1467] Processing CreateTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e (DEFAULT_TABLE table=test-table [id=47413a242e55435e83942658001ae82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:25.489305 31413 tablet_service.cc:1467] Processing CreateTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e (DEFAULT_TABLE table=test-table [id=47413a242e55435e83942658001ae82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:25.489333 31256 tablet_service.cc:1467] Processing CreateTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e (DEFAULT_TABLE table=test-table [id=47413a242e55435e83942658001ae82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:25.489423 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b86fa57e6c4501a6ec33b9fac0598e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.489434 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b86fa57e6c4501a6ec33b9fac0598e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.490540 31523 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Bootstrap starting.
I20260809 16:39:25.490837 31519 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Bootstrap starting.
I20260809 16:39:25.490983 31523 tablet_bootstrap.cc:654] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.491266 31519 tablet_bootstrap.cc:654] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.491972 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b86fa57e6c4501a6ec33b9fac0598e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:25.492183 31570 ts_tablet_manager.cc:1918] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:25.492245 31570 log.cc:1196] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/ab0379769a06483b950a73a4d5e3d8f3
I20260809 16:39:25.492508 31570 ts_tablet_manager.cc:1939] T ab0379769a06483b950a73a4d5e3d8f3 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting consensus metadata
I20260809 16:39:25.492904 30923 catalog_manager.cc:4841] TS 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085): tablet ab0379769a06483b950a73a4d5e3d8f3 (table test-table [id=2d35a151427d46e98576b33dad738436]) successfully deleted
I20260809 16:39:25.492962 31567 ts_tablet_manager.cc:1918] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:25.493026 31567 log.cc:1196] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/ab0379769a06483b950a73a4d5e3d8f3
I20260809 16:39:25.493312 31567 ts_tablet_manager.cc:1939] T ab0379769a06483b950a73a4d5e3d8f3 P 22a6158b551a4ea9823106a59860f5d2: Deleting consensus metadata
I20260809 16:39:25.493439 31517 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Bootstrap starting.
I20260809 16:39:25.493783 31519 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: No bootstrap required, opened a new log
I20260809 16:39:25.493830 31519 ts_tablet_manager.cc:1397] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260809 16:39:25.493887 31517 tablet_bootstrap.cc:654] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:25.493979 31519 raft_consensus.cc:348] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.494055 31519 raft_consensus.cc:374] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.494081 31519 raft_consensus.cc:729] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.494140 31519 consensus_queue.cc:260] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.494241 31519 ts_tablet_manager.cc:1428] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:25.494585 30924 catalog_manager.cc:4841] TS 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551): tablet ab0379769a06483b950a73a4d5e3d8f3 (table test-table [id=2d35a151427d46e98576b33dad738436]) successfully deleted
I20260809 16:39:25.494668 31517 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: No bootstrap required, opened a new log
I20260809 16:39:25.494704 31517 ts_tablet_manager.cc:1397] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:25.494869 31517 raft_consensus.cc:348] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.494932 31517 raft_consensus.cc:374] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.494952 31517 raft_consensus.cc:729] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.494993 31517 consensus_queue.cc:260] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.495132 31517 ts_tablet_manager.cc:1428] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:25.499147 31569 tablet_replica.cc:330] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: stopping tablet replica
I20260809 16:39:25.499285 31569 raft_consensus.cc:2229] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:25.499353 31569 raft_consensus.cc:2258] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:25.499536 31523 tablet_bootstrap.cc:492] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: No bootstrap required, opened a new log
I20260809 16:39:25.499591 31523 ts_tablet_manager.cc:1397] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260809 16:39:25.499722 31523 raft_consensus.cc:348] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.499756 31569 ts_tablet_manager.cc:1905] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:25.499768 31523 raft_consensus.cc:374] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:25.499922 31523 raft_consensus.cc:729] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Initialized, Role: FOLLOWER
I20260809 16:39:25.499979 31523 consensus_queue.cc:260] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.500097 31523 ts_tablet_manager.cc:1428] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:25.501322 31569 ts_tablet_manager.cc:1918] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:25.501398 31569 log.cc:1196] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/ab0379769a06483b950a73a4d5e3d8f3
I20260809 16:39:25.501652 31569 ts_tablet_manager.cc:1939] T ab0379769a06483b950a73a4d5e3d8f3 P 95950a97b93147f8b03dc76c12a814c0: Deleting consensus metadata
I20260809 16:39:25.502122 30921 catalog_manager.cc:4841] TS 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457): tablet ab0379769a06483b950a73a4d5e3d8f3 (table test-table [id=2d35a151427d46e98576b33dad738436]) successfully deleted
I20260809 16:39:25.571448 31526 raft_consensus.cc:482] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:25.571553 31526 raft_consensus.cc:504] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.571943 31526 leader_election.cc:288] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551)
I20260809 16:39:25.575593 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.572126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:39085 (local address 127.28.62.131:40585)
0809 16:39:25.572302 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.572312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.572363 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.572541 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.572542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.572544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:25.572627 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.572630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.573563 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.573563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.574603 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.574606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.575248 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.575250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.575353 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.575360 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.575446 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.575499 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260809 16:39:25.575614 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.572351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:40585 (local address 127.28.62.129:39085)
0809 16:39:25.572397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.572400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.572409 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:25.572440 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.572441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.572450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.572517 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:25.572722 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.573467 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.574741 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.575154 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.575237 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.575247 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.575486 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.575539 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:39:25.575661 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.572187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:59763 (local address 127.28.62.130:35551)
0809 16:39:25.572234 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.572236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.572344 (+ 108us) server_negotiation.cc:407] Connection header received
0809 16:39:25.572448 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.572449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.572466 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.572511 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:25.572735 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.573209 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.574655 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.575270 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.575358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.575367 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.575544 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.575591 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:25.575901 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.572120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.28.62.131:59763)
0809 16:39:25.572251 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.572263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.572368 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.572583 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.572583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.572585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:25.572644 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.572645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.573308 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.573309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.574544 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.574547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.575369 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.575369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.575457 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.575459 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.575747 (+ 288us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.575795 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":10,"thread_start_us":61,"threads_started":1}
I20260809 16:39:25.576073 31126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b86fa57e6c4501a6ec33b9fac0598e" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" is_pre_election: true
I20260809 16:39:25.576188 31126 raft_consensus.cc:2454] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 0.
I20260809 16:39:25.576244 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b86fa57e6c4501a6ec33b9fac0598e" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2" is_pre_election: true
I20260809 16:39:25.576393 31274 raft_consensus.cc:2454] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 0.
I20260809 16:39:25.576493 31360 leader_election.cc:302] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ddad79fcb6a4675800ebcb1898d66c6, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:25.576628 31526 raft_consensus.cc:2781] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:25.576675 31526 raft_consensus.cc:482] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:25.576702 31526 raft_consensus.cc:3037] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.577399 31526 raft_consensus.cc:504] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.577548 31526 leader_election.cc:288] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 election: Requested vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551)
I20260809 16:39:25.577687 31126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b86fa57e6c4501a6ec33b9fac0598e" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
I20260809 16:39:25.577773 31126 raft_consensus.cc:3037] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.577772 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b86fa57e6c4501a6ec33b9fac0598e" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2"
I20260809 16:39:25.577845 31274 raft_consensus.cc:3037] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:25.578593 31274 raft_consensus.cc:2454] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 1.
I20260809 16:39:25.578593 31126 raft_consensus.cc:2454] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 1.
I20260809 16:39:25.578800 31361 leader_election.cc:302] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:25.578904 31526 raft_consensus.cc:2781] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:25.579052 31526 raft_consensus.cc:686] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 1 LEADER]: Becoming Leader. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Running, Role: LEADER
I20260809 16:39:25.579123 31526 consensus_queue.cc:237] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:25.579864 30937 catalog_manager.cc:5495] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "95950a97b93147f8b03dc76c12a814c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } health_report { overall_health: HEALTHY } } }
I20260809 16:39:25.588501 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.585924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.28.62.131:38457)
0809 16:39:25.586091 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.586093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.586099 (+ 6us) server_negotiation.cc:407] Connection header received
0809 16:39:25.586130 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.586130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.586148 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.586217 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.586364 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.586827 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.587644 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.587870 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.587948 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.588113 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.588225 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.588332 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.588372 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.588391 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260809 16:39:25.593487 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.590991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:52251 (local address 127.28.62.130:35551)
0809 16:39:25.591075 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.591076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.591161 (+ 85us) server_negotiation.cc:407] Connection header received
0809 16:39:25.591226 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.591227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.591236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.591286 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.591401 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.591820 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.592846 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.593019 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.593070 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.593272 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.593301 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.593338 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.593422 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.593438 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 16:39:25.593551 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.590900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.28.62.131:52251)
0809 16:39:25.591097 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.591109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.591170 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.591286 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.591287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.591288 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:25.591347 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.591348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.591890 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.591890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.592765 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.592767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.593115 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.593116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.593175 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.593191 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.593364 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.593367 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.593420 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.593459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260809 16:39:25.595919 31126 raft_consensus.cc:1261] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Refusing update from remote peer 95950a97b93147f8b03dc76c12a814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.596045 31274 raft_consensus.cc:1261] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Refusing update from remote peer 95950a97b93147f8b03dc76c12a814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:25.596179 31578 consensus_queue.cc:1035] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.596292 31526 consensus_queue.cc:1035] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:25.604307 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.601701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:38457 (local address 127.28.62.130:40807)
0809 16:39:25.601750 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:25.601761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:25.601799 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:25.601987 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:25.601988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:25.601990 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:25.602031 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.602032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.602632 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.602634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.603529 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:25.603532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.603895 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.603896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:25.603949 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.603956 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.604147 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.604149 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:25.604202 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:25.604244 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 16:39:25.604305 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:25.601792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:40807 (local address 127.28.62.131:38457)
0809 16:39:25.601875 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:25.601877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:25.601884 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:25.601904 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:25.601904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:25.601914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:25.601963 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:25.602098 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.602553 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.603599 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:25.603809 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:25.603895 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:25.604046 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:25.604076 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:25.604146 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:25.604208 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 16:39:25.604226 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 16:39:25.604877 31548 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 16:39:27.125871 30938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:27.125985 30938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:27.126837 30938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Sending DeleteTablet for 3 replicas of tablet 80b86fa57e6c4501a6ec33b9fac0598e
I20260809 16:39:27.127192 31103 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37662
I20260809 16:39:27.127200 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50480
I20260809 16:39:27.127419 31413 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b86fa57e6c4501a6ec33b9fac0598e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33194
I20260809 16:39:27.127616 31617 tablet_replica.cc:330] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: stopping tablet replica
I20260809 16:39:27.127657 31616 tablet_replica.cc:330] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: stopping tablet replica
I20260809 16:39:27.127686 31617 raft_consensus.cc:2229] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:27.127722 31616 raft_consensus.cc:2229] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:27.127758 31616 raft_consensus.cc:2258] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:27.127835 31617 raft_consensus.cc:2258] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:27.128029 31616 ts_tablet_manager.cc:1905] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.128108 31617 ts_tablet_manager.cc:1905] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.129726 31616 ts_tablet_manager.cc:1918] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 16:39:27.129856 31616 log.cc:1196] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/80b86fa57e6c4501a6ec33b9fac0598e
I20260809 16:39:27.130049 31617 ts_tablet_manager.cc:1918] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 16:39:27.130160 31617 log.cc:1196] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/80b86fa57e6c4501a6ec33b9fac0598e
I20260809 16:39:27.130177 31616 ts_tablet_manager.cc:1939] T 80b86fa57e6c4501a6ec33b9fac0598e P 22a6158b551a4ea9823106a59860f5d2: Deleting consensus metadata
I20260809 16:39:27.130484 31617 ts_tablet_manager.cc:1939] T 80b86fa57e6c4501a6ec33b9fac0598e P 95950a97b93147f8b03dc76c12a814c0: Deleting consensus metadata
I20260809 16:39:27.130528 30924 catalog_manager.cc:4841] TS 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551): tablet 80b86fa57e6c4501a6ec33b9fac0598e (table test-table [id=47413a242e55435e83942658001ae82e]) successfully deleted
I20260809 16:39:27.130847 30921 catalog_manager.cc:4841] TS 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457): tablet 80b86fa57e6c4501a6ec33b9fac0598e (table test-table [id=47413a242e55435e83942658001ae82e]) successfully deleted
I20260809 16:39:27.132949 31614 tablet_replica.cc:330] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: stopping tablet replica
I20260809 16:39:27.133033 31614 raft_consensus.cc:2229] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:27.133082 31614 raft_consensus.cc:2258] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:27.133298 31614 ts_tablet_manager.cc:1905] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.134552 31614 ts_tablet_manager.cc:1918] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 16:39:27.134630 31614 log.cc:1196] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/80b86fa57e6c4501a6ec33b9fac0598e
I20260809 16:39:27.134840 31614 ts_tablet_manager.cc:1939] T 80b86fa57e6c4501a6ec33b9fac0598e P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting consensus metadata
I20260809 16:39:27.135210 30923 catalog_manager.cc:4841] TS 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085): tablet 80b86fa57e6c4501a6ec33b9fac0598e (table test-table [id=47413a242e55435e83942658001ae82e]) successfully deleted
Aug 09 16:39:27 dist-test-slave-wl2h krb5kdc[30892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 16:39:27 dist-test-slave-wl2h krb5kdc[30892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:27.157474 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.144750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.28.62.190:40287)
0809 16:39:27.144910 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.144912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.144923 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:27.144983 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.144983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.144994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.145062 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:27.145708 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.146225 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.146954 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.147209 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.147268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.154711 (+ 7443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:27.154713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:27.154713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:27.154720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:27.156432 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:27.156740 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:27.156741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:27.156742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:27.156811 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:27.156985 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:27.156986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:27.156986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:27.157133 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:27.157223 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.157332 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.157400 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260809 16:39:27.158845 30938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:27.159003 30938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:27.160470 31103 tablet_service.cc:1467] Processing CreateTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 (DEFAULT_TABLE table=test-table [id=a9b7089f29f44ba680a06d19519a1212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.160503 31413 tablet_service.cc:1467] Processing CreateTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 (DEFAULT_TABLE table=test-table [id=a9b7089f29f44ba680a06d19519a1212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.160490 31256 tablet_service.cc:1467] Processing CreateTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 (DEFAULT_TABLE table=test-table [id=a9b7089f29f44ba680a06d19519a1212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.160622 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3c564aaa2e49dbbd496b2e42919027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.160621 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3c564aaa2e49dbbd496b2e42919027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.160648 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3c564aaa2e49dbbd496b2e42919027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.162107 31630 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Bootstrap starting.
I20260809 16:39:27.162292 31631 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Bootstrap starting.
I20260809 16:39:27.162513 31630 tablet_bootstrap.cc:654] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.162690 31631 tablet_bootstrap.cc:654] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.162940 31629 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Bootstrap starting.
I20260809 16:39:27.163298 31631 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: No bootstrap required, opened a new log
I20260809 16:39:27.163302 31629 tablet_bootstrap.cc:654] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.163297 31630 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: No bootstrap required, opened a new log
I20260809 16:39:27.163367 31630 ts_tablet_manager.cc:1397] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:27.163370 31631 ts_tablet_manager.cc:1397] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:27.163494 31631 raft_consensus.cc:348] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.163614 31631 raft_consensus.cc:374] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.163656 31631 raft_consensus.cc:729] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.163730 31631 consensus_queue.cc:260] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.163724 31630 raft_consensus.cc:348] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.163842 31630 raft_consensus.cc:374] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.163880 31630 raft_consensus.cc:729] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.163939 31630 consensus_queue.cc:260] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.164012 31631 ts_tablet_manager.cc:1428] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:27.164141 31630 ts_tablet_manager.cc:1428] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:27.164700 31629 tablet_bootstrap.cc:492] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: No bootstrap required, opened a new log
I20260809 16:39:27.164803 31629 ts_tablet_manager.cc:1397] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:27.164921 31629 raft_consensus.cc:348] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.165014 31629 raft_consensus.cc:374] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.165035 31629 raft_consensus.cc:729] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.165087 31629 consensus_queue.cc:260] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.165191 31629 ts_tablet_manager.cc:1428] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:27.179203 31608 raft_consensus.cc:482] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:27.179313 31608 raft_consensus.cc:504] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.179476 31608 leader_election.cc:288] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551)
I20260809 16:39:27.179663 31126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc3c564aaa2e49dbbd496b2e42919027" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" is_pre_election: true
I20260809 16:39:27.179693 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc3c564aaa2e49dbbd496b2e42919027" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2" is_pre_election: true
I20260809 16:39:27.179785 31126 raft_consensus.cc:2454] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 0.
I20260809 16:39:27.179824 31274 raft_consensus.cc:2454] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 0.
I20260809 16:39:27.180018 31360 leader_election.cc:302] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ddad79fcb6a4675800ebcb1898d66c6, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:27.180140 31608 raft_consensus.cc:2781] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:27.180203 31608 raft_consensus.cc:482] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:27.180235 31608 raft_consensus.cc:3037] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.180998 31608 raft_consensus.cc:504] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.181150 31608 leader_election.cc:288] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 election: Requested vote from peers 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085), 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551)
I20260809 16:39:27.181294 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc3c564aaa2e49dbbd496b2e42919027" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2"
I20260809 16:39:27.181377 31274 raft_consensus.cc:3037] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.181365 31126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc3c564aaa2e49dbbd496b2e42919027" candidate_uuid: "95950a97b93147f8b03dc76c12a814c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddad79fcb6a4675800ebcb1898d66c6"
I20260809 16:39:27.181500 31126 raft_consensus.cc:3037] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.182238 31274 raft_consensus.cc:2454] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 1.
I20260809 16:39:27.182413 31126 raft_consensus.cc:2454] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95950a97b93147f8b03dc76c12a814c0 in term 1.
I20260809 16:39:27.182449 31361 leader_election.cc:302] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a6158b551a4ea9823106a59860f5d2, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:27.182602 31608 raft_consensus.cc:2781] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:27.182662 31608 raft_consensus.cc:686] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 1 LEADER]: Becoming Leader. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Running, Role: LEADER
I20260809 16:39:27.182734 31608 consensus_queue.cc:237] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.183449 30938 catalog_manager.cc:5495] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "95950a97b93147f8b03dc76c12a814c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } health_report { overall_health: HEALTHY } } }
I20260809 16:39:27.194123 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.191886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.28.62.131:38457)
0809 16:39:27.192068 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.192070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.192077 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:27.192095 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.192095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.192104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.192160 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:27.192298 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.192787 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.193534 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.193724 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.193789 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.193902 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.193927 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.193959 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.193993 (+ 34us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.194010 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260809 16:39:27.196257 31126 raft_consensus.cc:1261] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Refusing update from remote peer 95950a97b93147f8b03dc76c12a814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:27.196298 31274 raft_consensus.cc:1261] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Refusing update from remote peer 95950a97b93147f8b03dc76c12a814c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:27.196484 31608 consensus_queue.cc:1035] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:27.196623 31578 consensus_queue.cc:1035] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:27.206163 30938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:27.206290 30938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:27.206924 30938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Sending DeleteTablet for 3 replicas of tablet cc3c564aaa2e49dbbd496b2e42919027
I20260809 16:39:27.207242 31103 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37662
I20260809 16:39:27.207242 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50480
I20260809 16:39:27.207245 31413 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3c564aaa2e49dbbd496b2e42919027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:27 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33194
I20260809 16:39:27.207371 31617 tablet_replica.cc:330] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: stopping tablet replica
I20260809 16:39:27.207376 31616 tablet_replica.cc:330] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: stopping tablet replica
I20260809 16:39:27.207432 31617 raft_consensus.cc:2229] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:27.207444 31616 raft_consensus.cc:2229] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:27.207486 31616 raft_consensus.cc:2258] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:27.207485 31617 raft_consensus.cc:2258] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:27.207569 31614 tablet_replica.cc:330] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: stopping tablet replica
I20260809 16:39:27.207567 30938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 16:39:27.207631 31614 raft_consensus.cc:2229] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:27.207696 31614 raft_consensus.cc:2258] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 16:39:27.207712 30938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:27.208065 31614 ts_tablet_manager.cc:1905] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.208123 31617 ts_tablet_manager.cc:1905] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.208212 31616 ts_tablet_manager.cc:1905] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:27.209594 31617 ts_tablet_manager.cc:1918] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:27.209640 31617 log.cc:1196] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/cc3c564aaa2e49dbbd496b2e42919027
I20260809 16:39:27.209671 31614 ts_tablet_manager.cc:1918] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:27.209728 31614 log.cc:1196] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/cc3c564aaa2e49dbbd496b2e42919027
I20260809 16:39:27.209796 31616 ts_tablet_manager.cc:1918] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:27.209846 31616 log.cc:1196] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/cc3c564aaa2e49dbbd496b2e42919027
I20260809 16:39:27.209914 31617 ts_tablet_manager.cc:1939] T cc3c564aaa2e49dbbd496b2e42919027 P 95950a97b93147f8b03dc76c12a814c0: Deleting consensus metadata
I20260809 16:39:27.209977 31614 ts_tablet_manager.cc:1939] T cc3c564aaa2e49dbbd496b2e42919027 P 5ddad79fcb6a4675800ebcb1898d66c6: Deleting consensus metadata
I20260809 16:39:27.210073 31616 ts_tablet_manager.cc:1939] T cc3c564aaa2e49dbbd496b2e42919027 P 22a6158b551a4ea9823106a59860f5d2: Deleting consensus metadata
I20260809 16:39:27.210201 30921 catalog_manager.cc:4841] TS 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457): tablet cc3c564aaa2e49dbbd496b2e42919027 (table test-table [id=a9b7089f29f44ba680a06d19519a1212]) successfully deleted
I20260809 16:39:27.210343 30923 catalog_manager.cc:4841] TS 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085): tablet cc3c564aaa2e49dbbd496b2e42919027 (table test-table [id=a9b7089f29f44ba680a06d19519a1212]) successfully deleted
I20260809 16:39:27.210647 30924 catalog_manager.cc:4841] TS 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551): tablet cc3c564aaa2e49dbbd496b2e42919027 (table test-table [id=a9b7089f29f44ba680a06d19519a1212]) successfully deleted
I20260809 16:39:27.211202 31103 tablet_service.cc:1467] Processing CreateTablet for tablet e0c5daadec2d43848747912c25333210 (DEFAULT_TABLE table=test-table [id=18e100c03c3c472298e2d05e4ae87800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.211202 31256 tablet_service.cc:1467] Processing CreateTablet for tablet e0c5daadec2d43848747912c25333210 (DEFAULT_TABLE table=test-table [id=18e100c03c3c472298e2d05e4ae87800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.211313 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c5daadec2d43848747912c25333210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.211313 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c5daadec2d43848747912c25333210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.211299 31413 tablet_service.cc:1467] Processing CreateTablet for tablet e0c5daadec2d43848747912c25333210 (DEFAULT_TABLE table=test-table [id=18e100c03c3c472298e2d05e4ae87800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:27.211401 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c5daadec2d43848747912c25333210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:27.212551 31630 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6: Bootstrap starting.
I20260809 16:39:27.212659 31631 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Bootstrap starting.
I20260809 16:39:27.212942 31630 tablet_bootstrap.cc:654] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.212960 31631 tablet_bootstrap.cc:654] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.213192 31629 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Bootstrap starting.
I20260809 16:39:27.213537 31630 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6: No bootstrap required, opened a new log
I20260809 16:39:27.213549 31629 tablet_bootstrap.cc:654] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:27.213580 31630 ts_tablet_manager.cc:1397] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:27.213725 31630 raft_consensus.cc:348] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.213819 31630 raft_consensus.cc:374] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.213852 31630 raft_consensus.cc:729] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.213896 31630 consensus_queue.cc:260] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.214030 31630 ts_tablet_manager.cc:1428] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:27.214123 31631 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: No bootstrap required, opened a new log
I20260809 16:39:27.214186 31631 ts_tablet_manager.cc:1397] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:27.214324 31631 raft_consensus.cc:348] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.214401 31631 raft_consensus.cc:374] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.214425 31631 raft_consensus.cc:729] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a6158b551a4ea9823106a59860f5d2, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.214473 31631 consensus_queue.cc:260] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.214579 31631 ts_tablet_manager.cc:1428] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:27.214790 31629 tablet_bootstrap.cc:492] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: No bootstrap required, opened a new log
I20260809 16:39:27.214840 31629 ts_tablet_manager.cc:1397] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:27.214947 31629 raft_consensus.cc:348] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.215054 31629 raft_consensus.cc:374] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:27.215080 31629 raft_consensus.cc:729] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95950a97b93147f8b03dc76c12a814c0, State: Initialized, Role: FOLLOWER
I20260809 16:39:27.215140 31629 consensus_queue.cc:260] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.215266 31629 ts_tablet_manager.cc:1428] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:27.216192 31635 raft_consensus.cc:482] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:27.216252 31635 raft_consensus.cc:504] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.216526 31635 leader_election.cc:288] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551), 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457)
I20260809 16:39:27.220876 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.216808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47415 (local address 127.28.62.130:35551)
0809 16:39:27.216923 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.216925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.216951 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:39:27.217038 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.217039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.217052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.217140 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:27.217316 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.218144 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.219815 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.220482 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.220582 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.220596 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.220758 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.220806 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260809 16:39:27.220875 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.216759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:46011 (local address 127.28.62.131:38457)
0809 16:39:27.216805 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.216807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.217002 (+ 195us) server_negotiation.cc:407] Connection header received
0809 16:39:27.217098 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.217099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.217109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.217168 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:27.217328 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.218144 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.219869 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.220485 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.220605 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.220617 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.220759 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.220803 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:27.220896 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.216696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.28.62.129:47415)
0809 16:39:27.216887 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:27.216900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:27.216972 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:27.217142 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:27.217145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:27.217147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:27.217238 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.217240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.218255 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.218256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.219698 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.219702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.220583 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.220584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.220667 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.220675 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 16:39:27.220734 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:39:27.220787 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20260809 16:39:27.220896 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.216696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:38457 (local address 127.28.62.129:46011)
0809 16:39:27.216937 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:27.216948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:27.217030 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:27.217181 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:27.217182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:27.217184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:27.217250 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.217251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.218235 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.218236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.219698 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.219702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.220599 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.220600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.220667 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.220672 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:27.220734 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 16:39:27.220785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":79,"thread_start_us":115,"threads_started":1}
I20260809 16:39:27.221594 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c5daadec2d43848747912c25333210" candidate_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0" is_pre_election: true
I20260809 16:39:27.221691 31434 raft_consensus.cc:2454] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ddad79fcb6a4675800ebcb1898d66c6 in term 0.
I20260809 16:39:27.221712 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c5daadec2d43848747912c25333210" candidate_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2" is_pre_election: true
I20260809 16:39:27.221819 31274 raft_consensus.cc:2454] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ddad79fcb6a4675800ebcb1898d66c6 in term 0.
I20260809 16:39:27.221915 31046 leader_election.cc:302] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ddad79fcb6a4675800ebcb1898d66c6, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:27.222076 31635 raft_consensus.cc:2781] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:27.222126 31635 raft_consensus.cc:482] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:27.222158 31635 raft_consensus.cc:3037] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.222822 31635 raft_consensus.cc:504] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.222963 31635 leader_election.cc:288] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [CANDIDATE]: Term 1 election: Requested vote from peers 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551), 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457)
I20260809 16:39:27.223152 31434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c5daadec2d43848747912c25333210" candidate_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95950a97b93147f8b03dc76c12a814c0"
I20260809 16:39:27.223124 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c5daadec2d43848747912c25333210" candidate_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a6158b551a4ea9823106a59860f5d2"
I20260809 16:39:27.223248 31274 raft_consensus.cc:3037] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.223248 31434 raft_consensus.cc:3037] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:27.223928 31434 raft_consensus.cc:2454] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ddad79fcb6a4675800ebcb1898d66c6 in term 1.
I20260809 16:39:27.223985 31274 raft_consensus.cc:2454] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ddad79fcb6a4675800ebcb1898d66c6 in term 1.
I20260809 16:39:27.224112 31046 leader_election.cc:302] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ddad79fcb6a4675800ebcb1898d66c6, 95950a97b93147f8b03dc76c12a814c0; no voters:
I20260809 16:39:27.224207 31635 raft_consensus.cc:2781] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:27.224433 31635 raft_consensus.cc:686] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5ddad79fcb6a4675800ebcb1898d66c6, State: Running, Role: LEADER
I20260809 16:39:27.224521 31635 consensus_queue.cc:237] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } }
I20260809 16:39:27.225199 30938 catalog_manager.cc:5495] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129). New cstate: current_term: 1 leader_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddad79fcb6a4675800ebcb1898d66c6" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:27.233402 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.230948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.28.62.129:39085)
0809 16:39:27.231126 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.231128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.231138 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:27.231177 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.231178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.231199 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.231299 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:27.231411 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.231864 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.232664 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.232843 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.232894 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.233059 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.233157 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.233210 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.233251 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.233272 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260809 16:39:27.238119 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.235800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:36437 (local address 127.28.62.130:35551)
0809 16:39:27.235866 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.235868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.235894 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:39:27.235970 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.235970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.235980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.236019 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:27.236168 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.236635 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.237382 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.237582 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.237668 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.237865 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.237902 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.237964 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.238025 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.238047 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:39:27.238147 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.235647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35551 (local address 127.28.62.129:36437)
0809 16:39:27.235832 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:27.235845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:27.235910 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:27.236031 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:27.236032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:27.236033 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:27.236096 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.236097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.236720 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.236722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.237293 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.237295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.237672 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.237673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.237748 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.237762 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.237966 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.237969 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:27.238021 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:39:27.238062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260809 16:39:27.240243 31274 raft_consensus.cc:1261] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Refusing update from remote peer 5ddad79fcb6a4675800ebcb1898d66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:27.240388 31434 raft_consensus.cc:1261] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Refusing update from remote peer 5ddad79fcb6a4675800ebcb1898d66c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:27.240471 31635 consensus_queue.cc:1035] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a6158b551a4ea9823106a59860f5d2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:27.240614 31649 consensus_queue.cc:1035] T e0c5daadec2d43848747912c25333210 P 5ddad79fcb6a4675800ebcb1898d66c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95950a97b93147f8b03dc76c12a814c0" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:27.248452 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.245814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:35429 (local address 127.28.62.129:39085)
0809 16:39:27.245850 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:27.245853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:27.245995 (+ 142us) server_negotiation.cc:407] Connection header received
0809 16:39:27.246101 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:27.246102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:27.246116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:27.246175 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:27.246316 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.246768 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.247778 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.248064 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.248116 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.248259 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.248292 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.248324 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:27.248367 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 16:39:27.248382 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 16:39:27.248457 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:27.245769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:39085 (local address 127.28.62.130:35429)
0809 16:39:27.245935 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:27.245947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:27.246021 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:27.246193 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:27.246194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:27.246196 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:27.246249 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.246251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.246831 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.246832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.247629 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:27.247632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:27.248153 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:27.248154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:27.248199 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:27.248203 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:27.248334 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:27.248335 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:27.248366 (+ 31us) client_negotiation.cc:240] Negotiation successful
0809 16:39:27.248400 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":116,"threads_started":1}
W20260809 16:39:27.248831 31595 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 16:39:28.768050 30938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:28.768174 30938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:28.769127 30938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c30d4b390f2c4ad5af64faa94aa84b2a: Sending DeleteTablet for 3 replicas of tablet e0c5daadec2d43848747912c25333210
I20260809 16:39:28.769459 31413 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c5daadec2d43848747912c25333210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:28 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33194
I20260809 16:39:28.769459 31103 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c5daadec2d43848747912c25333210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:28 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37662
I20260809 16:39:28.769477 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c5daadec2d43848747912c25333210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:28 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50480
I20260809 16:39:28.769685 31692 tablet_replica.cc:330] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: stopping tablet replica
I20260809 16:39:28.769773 31692 raft_consensus.cc:2229] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:28.769812 31692 raft_consensus.cc:2258] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:28.769791 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31027
I20260809 16:39:28.773188 31692 ts_tablet_manager.cc:1905] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:28.775439 31692 ts_tablet_manager.cc:1918] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 16:39:28.775506 31692 log.cc:1196] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/e0c5daadec2d43848747912c25333210
I20260809 16:39:28.775712 31692 ts_tablet_manager.cc:1939] T e0c5daadec2d43848747912c25333210 P 22a6158b551a4ea9823106a59860f5d2: Deleting consensus metadata
I20260809 16:39:28.776145 30924 catalog_manager.cc:4841] TS 22a6158b551a4ea9823106a59860f5d2 (127.28.62.130:35551): tablet e0c5daadec2d43848747912c25333210 (table test-table [id=18e100c03c3c472298e2d05e4ae87800]) successfully deleted
I20260809 16:39:28.776680 31690 tablet_replica.cc:330] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: stopping tablet replica
I20260809 16:39:28.776772 31690 raft_consensus.cc:2229] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:28.776842 31690 raft_consensus.cc:2258] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:28.777110 31690 ts_tablet_manager.cc:1905] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:28.779074 31690 ts_tablet_manager.cc:1918] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 16:39:28.779145 31690 log.cc:1196] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/e0c5daadec2d43848747912c25333210
I20260809 16:39:28.779371 31690 ts_tablet_manager.cc:1939] T e0c5daadec2d43848747912c25333210 P 95950a97b93147f8b03dc76c12a814c0: Deleting consensus metadata
I20260809 16:39:28.779906 30921 catalog_manager.cc:4841] TS 95950a97b93147f8b03dc76c12a814c0 (127.28.62.131:38457): tablet e0c5daadec2d43848747912c25333210 (table test-table [id=18e100c03c3c472298e2d05e4ae87800]) successfully deleted
W20260809 16:39:28.782995 30923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:39085 (error 108)
I20260809 16:39:28.783255 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31187
I20260809 16:39:28.786136 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:28.783455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:39085 (local address 127.0.0.1:36310)
0809 16:39:28.786023 (+ 2568us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:28.786083 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2538,"thread_start_us":67,"threads_started":1}
W20260809 16:39:28.786296 30923 catalog_manager.cc:4584] TS 5ddad79fcb6a4675800ebcb1898d66c6 (127.28.62.129:39085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0c5daadec2d43848747912c25333210: Network error: Client connection negotiation failed: client connection to 127.28.62.129:39085: connect: Connection refused (error 111)
I20260809 16:39:28.792995 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31341
I20260809 16:39:28.802073 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 30905
2026-08-09T16:39:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5842 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:28 dist-test-slave-wl2h krb5kdc[31698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:28 dist-test-slave-wl2h krb5kdc[31698](info): set up 2 sockets
Aug 09 16:39:28 dist-test-slave-wl2h krb5kdc[31698](info): commencing operation
krb5kdc: starting...
W20260809 16:39:29.950346 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:30.094851 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46337
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:30.181716 31711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:30.181955 31711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:30.181993 31711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:30.183157 31711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:30.183238 31711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:30.183262 31711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:30.183290 31711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:30.183320 31711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:30.184755 31711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:30.185027 31711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:30.185269 31711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:30.190223 31717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.190243 31720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.190219 31718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.190583 31711 server_base.cc:1034] running on GCE node
I20260809 16:39:30.190847 31711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:30.191231 31711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:30.192397 31711 hybrid_clock.cc:648] HybridClock initialized: now 1786293570192375 us; error 46 us; skew 500 ppm
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:30.194876 31711 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:30.195358 31711 webserver.cc:460] Webserver started at http://127.28.62.190:39325/ using document root <none> and password file <none>
I20260809 16:39:30.195588 31711 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:30.195655 31711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:30.195797 31711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:30.196837 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "74bf68781017480787c0c2d5af2cc7dc"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "28385df0d88d26f43b31a82a259dfa29"
server_key_iv: "36acf29d9a9b1ef8e30e16866a892fc9"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.197221 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "74bf68781017480787c0c2d5af2cc7dc"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "28385df0d88d26f43b31a82a259dfa29"
server_key_iv: "36acf29d9a9b1ef8e30e16866a892fc9"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.198647 31711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:30.199471 31729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.199640 31711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.199746 31711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "74bf68781017480787c0c2d5af2cc7dc"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "28385df0d88d26f43b31a82a259dfa29"
server_key_iv: "36acf29d9a9b1ef8e30e16866a892fc9"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.199828 31711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:30.210644 31711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:30.212543 31711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:30.212713 31711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:30.217128 31711 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:46337
I20260809 16:39:30.217167 31786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:46337 every 8 connection(s)
I20260809 16:39:30.217514 31711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:30.218389 31788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:30.220564 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31711
I20260809 16:39:30.220671 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:30.220862 28922 external_mini_cluster.cc:1428] Setting key 021277daf2a70cde111b82000fb7d003
I20260809 16:39:30.220875 31788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc: Bootstrap starting.
I20260809 16:39:30.221532 31788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:30.221859 31788 log.cc:824] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:30.222556 31788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc: No bootstrap required, opened a new log
I20260809 16:39:30.224421 31788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } }
I20260809 16:39:30.224581 31788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:30.224648 31788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bf68781017480787c0c2d5af2cc7dc, State: Initialized, Role: FOLLOWER
I20260809 16:39:30.224812 31788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } }
I20260809 16:39:30.224900 31788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:30.224946 31788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:30.224994 31788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:30.225531 31788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } }
I20260809 16:39:30.225694 31788 leader_election.cc:302] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74bf68781017480787c0c2d5af2cc7dc; no voters:
I20260809 16:39:30.225870 31788 leader_election.cc:288] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:30.225899 31793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:30.226018 31793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [term 1 LEADER]: Becoming Leader. State: Replica: 74bf68781017480787c0c2d5af2cc7dc, State: Running, Role: LEADER
I20260809 16:39:30.226138 31788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:30.226121 31793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } }
I20260809 16:39:30.226529 31796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74bf68781017480787c0c2d5af2cc7dc. Latest consensus state: current_term: 1 leader_uuid: "74bf68781017480787c0c2d5af2cc7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } } }
I20260809 16:39:30.226691 31796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:30.227286 31803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:30.227807 31803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:30.227782 31794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74bf68781017480787c0c2d5af2cc7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bf68781017480787c0c2d5af2cc7dc" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46337 } } }
I20260809 16:39:30.227954 31794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc [sys.catalog]: This master's current role is: LEADER
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:30.229954 31803 catalog_manager.cc:1347] Generated new cluster ID: d2ec0cb7b1334a15a2d8f563cd8e705b
I20260809 16:39:30.230005 31803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:30.232859 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.221423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.28.62.190:46337)
0809 16:39:30.221632 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:30.221641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:30.221675 (+ 34us) server_negotiation.cc:407] Connection header received
0809 16:39:30.222199 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:30.222205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:30.222251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:30.222402 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:30.223118 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.223997 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.224916 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.225399 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.225532 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.228816 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:30.228821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:30.228821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:30.228835 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:30.230710 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.231183 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.231186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.231187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.231298 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.231524 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.231525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.231526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.231825 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:30.231978 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:30.232507 (+ 529us) server_negotiation.cc:299] Negotiation successful
0809 16:39:30.232647 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260809 16:39:30.237509 31803 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:30.237994 31803 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:30.242417 31803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74bf68781017480787c0c2d5af2cc7dc: Generated new TSK 0
I20260809 16:39:30.242571 31803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:30.275455 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:30.357533 31817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:30.357774 31817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:30.357815 31817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:30.358955 31817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:30.359019 31817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:30.359091 31817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:30.360520 31817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:30.360780 31817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:30.361009 31817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:30.366112 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.366201 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.366130 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.366410 31817 server_base.cc:1034] running on GCE node
I20260809 16:39:30.366611 31817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:30.366850 31817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:30.368013 31817 hybrid_clock.cc:648] HybridClock initialized: now 1786293570367979 us; error 71 us; skew 500 ppm
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:30.370497 31817 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:30.370870 31817 webserver.cc:460] Webserver started at http://127.28.62.129:36891/ using document root <none> and password file <none>
I20260809 16:39:30.371122 31817 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:30.371172 31817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:30.371309 31817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:30.372385 31817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "2d7c985c0bc6405197e45c06f53f3749"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "a2cff36f0cf9a0c80ce2b2620119acc6"
server_key_iv: "ef9c70f745f2282f79b41d03a7f275a5"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.372753 31817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "2d7c985c0bc6405197e45c06f53f3749"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "a2cff36f0cf9a0c80ce2b2620119acc6"
server_key_iv: "ef9c70f745f2282f79b41d03a7f275a5"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.374092 31817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:30.374912 31835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.375111 31817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:30.375211 31817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "2d7c985c0bc6405197e45c06f53f3749"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "a2cff36f0cf9a0c80ce2b2620119acc6"
server_key_iv: "ef9c70f745f2282f79b41d03a7f275a5"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.375379 31817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:30.386686 31817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:30.388672 31817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:30.388783 31817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:30.388968 31817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:30.389292 31817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:30.389346 31817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.389387 31817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:30.389410 31817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.395555 31817 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:44219
I20260809 16:39:30.395609 31959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:44219 every 8 connection(s)
I20260809 16:39:30.395946 31817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:30.401048 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31817
I20260809 16:39:30.401149 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:30.401335 28922 external_mini_cluster.cc:1428] Setting key 88e5d94526d38ae226c898482b3386ec
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:30.405830 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.396614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46337 (local address 127.28.62.129:38339)
0809 16:39:30.396793 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:30.396835 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:30.397313 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:30.397532 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:30.397535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:30.398096 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:30.398280 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.398285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.398943 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.398944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.399788 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.399790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.400183 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.400184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.400282 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.400768 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:30.400777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:30.402576 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:30.404374 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.404381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.404384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.404707 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.404908 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.404908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.404909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.405060 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.405350 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:30.405356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:30.405469 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 16:39:30.405556 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 16:39:30.405675 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260809 16:39:30.405859 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.396655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:38339 (local address 127.28.62.190:46337)
0809 16:39:30.396723 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:30.396725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:30.396948 (+ 223us) server_negotiation.cc:407] Connection header received
0809 16:39:30.397411 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:30.397412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:30.397421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:30.397483 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:30.398361 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.398862 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.399866 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.400063 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.400175 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.402675 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:30.402677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:30.402677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:30.402685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:30.404290 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.404792 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.404793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.404793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.404847 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.405137 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.405138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.405138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.405275 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:30.405381 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:30.405598 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 16:39:30.405697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:30.406379 31960 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46337
I20260809 16:39:30.406497 31960 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:30.406672 31960 heartbeater.cc:511] Master 127.28.62.190:46337 requested a full tablet report, sending...
I20260809 16:39:30.407222 31747 ts_manager.cc:194] Registered new tserver with Master: 2d7c985c0bc6405197e45c06f53f3749 (127.28.62.129:44219)
I20260809 16:39:30.407828 31747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:38339
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:30.430388 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:30.512368 31965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:30.512578 31965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:30.512601 31965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:30.513751 31965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:30.513794 31965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:30.513821 31965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:30.515374 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:30.515617 31965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:30.515865 31965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:30.521032 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.521040 31975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.521111 31965 server_base.cc:1034] running on GCE node
W20260809 16:39:30.521052 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.521559 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:30.521819 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:30.522969 31965 hybrid_clock.cc:648] HybridClock initialized: now 1786293570522934 us; error 51 us; skew 500 ppm
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:30.525662 31965 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:30.526185 31965 webserver.cc:460] Webserver started at http://127.28.62.130:45307/ using document root <none> and password file <none>
I20260809 16:39:30.526429 31965 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:30.526499 31965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:30.526639 31965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:30.527871 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "16bdffdd05b845128c8a00c139903331"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "7816f1765c4ac7f8db0ffedb51cab63f"
server_key_iv: "7dfba70e902a25f8d1328e34fee716a3"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.528285 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "16bdffdd05b845128c8a00c139903331"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "7816f1765c4ac7f8db0ffedb51cab63f"
server_key_iv: "7dfba70e902a25f8d1328e34fee716a3"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.529644 31965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:39:30.530467 31984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.530694 31965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:30.530814 31965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "16bdffdd05b845128c8a00c139903331"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "7816f1765c4ac7f8db0ffedb51cab63f"
server_key_iv: "7dfba70e902a25f8d1328e34fee716a3"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.530898 31965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:30.545850 31965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:30.547792 31965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:30.547956 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:30.548206 31965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:30.548520 31965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:30.548575 31965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.548611 31965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:30.548671 31965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.554606 31965 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:34043
I20260809 16:39:30.554658 32107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:34043 every 8 connection(s)
I20260809 16:39:30.554972 31965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:30.555496 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 31965
I20260809 16:39:30.555603 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:30.555792 28922 external_mini_cluster.cc:1428] Setting key 523cdb5c7660edd2f125d4f17be09c15
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:39:30.566103 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.555889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:33289 (local address 127.28.62.190:46337)
0809 16:39:30.555940 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:30.555942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:30.556017 (+ 75us) server_negotiation.cc:407] Connection header received
0809 16:39:30.556754 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:30.556755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:30.556767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:30.556837 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:30.558211 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.558933 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.559984 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.560197 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.560284 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.563101 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:30.563103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:30.563103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:30.563112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:30.564529 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.565061 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.565062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.565063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.565131 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.565434 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.565435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.565435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.565542 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:30.565603 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:30.565966 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 16:39:30.566033 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:39:30.566380 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.555727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46337 (local address 127.28.62.130:33289)
0809 16:39:30.555895 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:30.555936 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:30.556626 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:30.556972 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:30.556976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:30.557823 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:30.558094 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.558100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.559078 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.559079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.559875 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.559877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.560296 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.560297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.560416 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.561004 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:30.561024 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:30.562940 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:30.564619 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.564626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.564628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.564954 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.565218 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.565219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.565219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.565363 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.565618 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:30.565624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:30.565805 (+ 181us) client_negotiation.cc:769] Sending connection context
0809 16:39:30.565932 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 16:39:30.566113 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260809 16:39:30.566928 32108 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46337
I20260809 16:39:30.567063 32108 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:30.567232 32108 heartbeater.cc:511] Master 127.28.62.190:46337 requested a full tablet report, sending...
I20260809 16:39:30.567607 31747 ts_manager.cc:194] Registered new tserver with Master: 16bdffdd05b845128c8a00c139903331 (127.28.62.130:34043)
I20260809 16:39:30.567994 31747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:33289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:30.585968 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:30.668062 32114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:30.668289 32114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:30.668314 32114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:30.669523 32114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:30.669567 32114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:30.669595 32114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:30.670994 32114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:30.671226 32114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:30.671409 32114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:30.676445 32119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:30.676452 32122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.676806 32114 server_base.cc:1034] running on GCE node
W20260809 16:39:30.676452 32124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:30.677130 32114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:30.677333 32114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:30.678498 32114 hybrid_clock.cc:648] HybridClock initialized: now 1786293570678489 us; error 39 us; skew 500 ppm
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:30.681140 32114 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:30.681622 32114 webserver.cc:460] Webserver started at http://127.28.62.131:36323/ using document root <none> and password file <none>
I20260809 16:39:30.681872 32114 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:30.681942 32114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:30.682089 32114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:30.683202 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "75c8fbdbaf36412c9100a4a7984cce5d"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "d27705a481968cd76ac87bbe348bd64d"
server_key_iv: "01dda9072ab631a97b8ed04506a045e7"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.683550 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "75c8fbdbaf36412c9100a4a7984cce5d"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "d27705a481968cd76ac87bbe348bd64d"
server_key_iv: "01dda9072ab631a97b8ed04506a045e7"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.684931 32114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:39:30.685683 32132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.685854 32114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.685956 32114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "75c8fbdbaf36412c9100a4a7984cce5d"
format_stamp: "Formatted at 2026-08-09 16:39:30 on dist-test-slave-wl2h"
server_key: "d27705a481968cd76ac87bbe348bd64d"
server_key_iv: "01dda9072ab631a97b8ed04506a045e7"
server_key_version: "encryptionkey@0"
I20260809 16:39:30.686039 32114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:30.699380 32114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:30.701807 32114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:30.701998 32114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:30.702257 32114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:30.702589 32114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:30.702643 32114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.702690 32114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:30.702718 32114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:30.708803 32114 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:42669
I20260809 16:39:30.708854 32264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:42669 every 8 connection(s)
I20260809 16:39:30.709163 32114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:30.711447 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 32114
I20260809 16:39:30.711570 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:30.711752 28922 external_mini_cluster.cc:1428] Setting key f85d2f8eabbca6fd40e251941ea1fc67
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[31698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:30.719067 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.709888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:52693 (local address 127.28.62.190:46337)
0809 16:39:30.709967 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:30.709970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:30.710171 (+ 201us) server_negotiation.cc:407] Connection header received
0809 16:39:30.710705 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:30.710706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:30.710718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:30.710774 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:30.711916 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.712559 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.713578 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.713794 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.713873 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.716116 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:30.716117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:30.716117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:30.716126 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:30.717640 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.718050 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.718051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.718052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.718117 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:30.718369 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:30.718370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:30.718371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:30.718493 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:30.718570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:30.718849 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 16:39:30.718929 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 16:39:30.719260 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:30.709865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46337 (local address 127.28.62.131:52693)
0809 16:39:30.710030 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:30.710073 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:30.710592 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:30.710846 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:30.710850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:30.711573 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:30.711809 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.711814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.712651 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.712652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.713484 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:30.713486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:30.713881 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:30.713882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:30.714021 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:30.714563 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:30.714571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:30.716018 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:30.717743 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.717751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.717753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.717971 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.718186 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:30.718187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:30.718187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:30.718299 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:30.718561 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:30.718568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:30.718710 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 16:39:30.718816 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 16:39:30.718974 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260809 16:39:30.719834 32266 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46337
I20260809 16:39:30.719959 32266 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:30.720172 32266 heartbeater.cc:511] Master 127.28.62.190:46337 requested a full tablet report, sending...
I20260809 16:39:30.720534 31747 ts_manager.cc:194] Registered new tserver with Master: 75c8fbdbaf36412c9100a4a7984cce5d (127.28.62.131:42669)
I20260809 16:39:30.720908 31747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:52693
I20260809 16:39:30.723639 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 16:39:30.737443 31791 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:30.736626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.28.62.190:46337)
0809 16:39:30.736692 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:30.736694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:30.736774 (+ 80us) server_negotiation.cc:407] Connection header received
0809 16:39:30.736821 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:30.736822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:30.736830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:30.736888 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:30.737352 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41676: BlockingRecv error: recv got EOF from 127.0.0.1:41676 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 16:39:30.738384 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31817
I20260809 16:39:30.744427 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31965
I20260809 16:39:30.754617 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 32114
I20260809 16:39:30.760607 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 31711
2026-08-09T16:39:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1958 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[32279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[32279](info): set up 2 sockets
Aug 09 16:39:30 dist-test-slave-wl2h krb5kdc[32279](info): commencing operation
krb5kdc: starting...
W20260809 16:39:31.894322 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:31 dist-test-slave-wl2h krb5kdc[32279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:32.023864 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44309
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:32.105975 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:32.106184 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:32.106213 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:32.107376 32293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:32.107419 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:32.107434 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:32.107446 32293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:32.107501 32293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:32.108951 32293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:32.109210 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:32.109437 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:32.114364 32302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:32.114413 32293 server_base.cc:1034] running on GCE node
W20260809 16:39:32.114378 32304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:32.114392 32301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:32.114780 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:32.115167 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:32.116349 32293 hybrid_clock.cc:648] HybridClock initialized: now 1786293572116358 us; error 52 us; skew 500 ppm
Aug 09 16:39:32 dist-test-slave-wl2h krb5kdc[32279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:32.119089 32293 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:32.119549 32293 webserver.cc:460] Webserver started at http://127.28.62.190:43961/ using document root <none> and password file <none>
I20260809 16:39:32.119783 32293 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:32.119851 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:32.119987 32293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:32.121042 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "b0810aa189db403cb2a2df3b676da1ac"
format_stamp: "Formatted at 2026-08-09 16:39:32 on dist-test-slave-wl2h"
server_key: "30cda33ecd6c05715f6243d28dea4c76"
server_key_iv: "0947f4428c17d958acfaa17d3f965a20"
server_key_version: "encryptionkey@0"
I20260809 16:39:32.121464 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "b0810aa189db403cb2a2df3b676da1ac"
format_stamp: "Formatted at 2026-08-09 16:39:32 on dist-test-slave-wl2h"
server_key: "30cda33ecd6c05715f6243d28dea4c76"
server_key_iv: "0947f4428c17d958acfaa17d3f965a20"
server_key_version: "encryptionkey@0"
I20260809 16:39:32.122843 32293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:32.123695 32312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:32.123888 32293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:32.123991 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "b0810aa189db403cb2a2df3b676da1ac"
format_stamp: "Formatted at 2026-08-09 16:39:32 on dist-test-slave-wl2h"
server_key: "30cda33ecd6c05715f6243d28dea4c76"
server_key_iv: "0947f4428c17d958acfaa17d3f965a20"
server_key_version: "encryptionkey@0"
I20260809 16:39:32.124079 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:32.146716 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:32.148654 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:32.148835 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:32.153252 32293 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44309
I20260809 16:39:32.153288 32369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44309 every 8 connection(s)
I20260809 16:39:32.153848 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:32.154497 32371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:32.156934 32371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac: Bootstrap starting.
I20260809 16:39:32.157552 32371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:32.157866 32371 log.cc:824] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:32.158634 32371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac: No bootstrap required, opened a new log
I20260809 16:39:32.159518 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 32293
I20260809 16:39:32.159633 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:32.159814 28922 external_mini_cluster.cc:1428] Setting key 1ae78914e7462f5b754869f8a7c0665c
I20260809 16:39:32.161170 32371 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } }
I20260809 16:39:32.161332 32371 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:32.161397 32371 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0810aa189db403cb2a2df3b676da1ac, State: Initialized, Role: FOLLOWER
I20260809 16:39:32.161535 32371 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } }
I20260809 16:39:32.161630 32371 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:32.161672 32371 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:32.161723 32371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:32.162308 32371 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } }
I20260809 16:39:32.162431 32371 leader_election.cc:302] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0810aa189db403cb2a2df3b676da1ac; no voters:
I20260809 16:39:32.162639 32371 leader_election.cc:288] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:32.162676 32377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:32.162788 32377 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [term 1 LEADER]: Becoming Leader. State: Replica: b0810aa189db403cb2a2df3b676da1ac, State: Running, Role: LEADER
I20260809 16:39:32.162910 32371 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:32.162936 32377 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } }
I20260809 16:39:32.163308 32377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0810aa189db403cb2a2df3b676da1ac. Latest consensus state: current_term: 1 leader_uuid: "b0810aa189db403cb2a2df3b676da1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } } }
I20260809 16:39:32.163322 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0810aa189db403cb2a2df3b676da1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0810aa189db403cb2a2df3b676da1ac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44309 } } }
I20260809 16:39:32.163471 32377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:32.163704 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:32.163913 32385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:32.164449 32385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:32.166051 32385 catalog_manager.cc:1347] Generated new cluster ID: 2de6e5b423db453fbc4f1365813cd9db
I20260809 16:39:32.166105 32385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:39:32 dist-test-slave-wl2h krb5kdc[32279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:32.172181 32385 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:32.172978 32385 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:32.175534 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:32.160330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.28.62.190:44309)
0809 16:39:32.160517 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:32.160524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:32.160549 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:39:32.161159 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:32.161166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:32.161238 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:32.161391 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:32.162181 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:32.163462 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:32.164542 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:32.165289 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:32.165413 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:32.171460 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:32.171465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:32.171465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:32.171478 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:32.173632 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:32.174199 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:32.174202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:32.174202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:32.174308 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:32.174536 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:32.174537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:32.174537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:32.174805 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:32.174968 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:32.175118 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 16:39:32.175263 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260809 16:39:32.176460 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:39:32.179327 32385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0810aa189db403cb2a2df3b676da1ac: Generated new TSK 0
I20260809 16:39:32.179508 32385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:39:32.195672 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:32.191671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.28.62.190:44309)
0809 16:39:32.191730 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:32.191732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:32.191832 (+ 100us) server_negotiation.cc:407] Connection header received
0809 16:39:32.191900 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:32.191901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:32.191911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:32.191962 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:32.192443 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:32.192965 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:32.194145 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:32.194347 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:32.194402 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:32.195195 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:32.195196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:32.195197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:32.195397 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:32.195412 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:32.195418 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:32.195466 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:32.195565 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 16:39:32.195589 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:39:32.198458 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 32293
2026-08-09T16:39:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1437 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:32 dist-test-slave-wl2h krb5kdc[32410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:32 dist-test-slave-wl2h krb5kdc[32410](info): set up 2 sockets
Aug 09 16:39:32 dist-test-slave-wl2h krb5kdc[32410](info): commencing operation
krb5kdc: starting...
W20260809 16:39:33.322579 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:33.454146 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40459
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:33.536104 32424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:33.536355 32424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:33.536379 32424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:33.537448 32424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:33.537483 32424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:33.537496 32424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:33.537509 32424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:33.537521 32424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:33.539001 32424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:40459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:33.539204 32424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:33.539377 32424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:33.544166 32433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:33.544217 32430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:33.544220 32431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:33.544438 32424 server_base.cc:1034] running on GCE node
I20260809 16:39:33.544675 32424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:33.544912 32424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:33.546077 32424 hybrid_clock.cc:648] HybridClock initialized: now 1786293573546066 us; error 41 us; skew 500 ppm
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:33.548652 32424 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:33.549170 32424 webserver.cc:460] Webserver started at http://127.28.62.190:45543/ using document root <none> and password file <none>
I20260809 16:39:33.549489 32424 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:33.549552 32424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:33.549691 32424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:33.550850 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "0120db6a26e94b46935f273d6217dd50"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "2d088e34f8b00b0a90ef51098039192c"
server_key_iv: "cd5ec8b5eb06aa4e737ebf06e7e70734"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.551262 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "0120db6a26e94b46935f273d6217dd50"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "2d088e34f8b00b0a90ef51098039192c"
server_key_iv: "cd5ec8b5eb06aa4e737ebf06e7e70734"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.552692 32424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:33.553501 32441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.553709 32424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:33.553817 32424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "0120db6a26e94b46935f273d6217dd50"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "2d088e34f8b00b0a90ef51098039192c"
server_key_iv: "cd5ec8b5eb06aa4e737ebf06e7e70734"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.553890 32424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:33.569355 32424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:33.571744 32424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:33.571925 32424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:33.576265 32424 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:40459
I20260809 16:39:33.576303 32499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:40459 every 8 connection(s)
I20260809 16:39:33.576651 32424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:33.577616 32500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:33.579185 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 32424
I20260809 16:39:33.579270 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:33.579452 28922 external_mini_cluster.cc:1428] Setting key 0722a41ed29a2120bac57b23aa133306
I20260809 16:39:33.580547 32500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50: Bootstrap starting.
I20260809 16:39:33.581266 32500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:33.581598 32500 log.cc:824] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:33.582384 32500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50: No bootstrap required, opened a new log
I20260809 16:39:33.584757 32500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } }
I20260809 16:39:33.584964 32500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:33.585004 32500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0120db6a26e94b46935f273d6217dd50, State: Initialized, Role: FOLLOWER
I20260809 16:39:33.585162 32500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } }
I20260809 16:39:33.585269 32500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:33.585305 32500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:33.585361 32500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:33.585932 32500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } }
I20260809 16:39:33.586018 32500 leader_election.cc:302] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0120db6a26e94b46935f273d6217dd50; no voters:
I20260809 16:39:33.586203 32500 leader_election.cc:288] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:33.586236 32505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:33.586356 32505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [term 1 LEADER]: Becoming Leader. State: Replica: 0120db6a26e94b46935f273d6217dd50, State: Running, Role: LEADER
I20260809 16:39:33.586473 32500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:33.586498 32505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } }
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:33.586828 32505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0120db6a26e94b46935f273d6217dd50. Latest consensus state: current_term: 1 leader_uuid: "0120db6a26e94b46935f273d6217dd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } } }
I20260809 16:39:33.586917 32505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:33.586880 32506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0120db6a26e94b46935f273d6217dd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0120db6a26e94b46935f273d6217dd50" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40459 } } }
I20260809 16:39:33.587149 32506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:33.587174 32511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:33.587797 32511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:33.589512 32511 catalog_manager.cc:1347] Generated new cluster ID: 347569ac47334b05aa3f1c35916cdf3b
I20260809 16:39:33.589591 32511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:33.591004 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:33.579990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.28.62.190:40459)
0809 16:39:33.580198 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:33.580207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:33.580249 (+ 42us) server_negotiation.cc:407] Connection header received
0809 16:39:33.580772 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:33.580780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:33.580843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:33.581019 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:33.581630 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.582971 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.583915 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.584617 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.584729 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:33.587401 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:33.587405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:33.587406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:33.587418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:33.589398 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.589839 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.589842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.589842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.589940 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.590124 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.590125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.590125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.590402 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:33.590539 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:33.590680 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 16:39:33.590815 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260809 16:39:33.600190 32511 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:33.600724 32511 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:33.619451 32511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0120db6a26e94b46935f273d6217dd50: Generated new TSK 0
I20260809 16:39:33.619666 32511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:33.648243 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:33.730813 32527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:33.731032 32527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:33.731057 32527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:33.732208 32527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:33.732249 32527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:33.732275 32527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:33.733713 32527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:33.733896 32527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:33.734072 32527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:33.739077 32535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:33.739156 32527 server_base.cc:1034] running on GCE node
W20260809 16:39:33.739066 32534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:33.739976 32539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:33.740224 32527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:33.740474 32527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:33.741638 32527 hybrid_clock.cc:648] HybridClock initialized: now 1786293573741630 us; error 39 us; skew 500 ppm
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:33.744136 32527 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:33.744591 32527 webserver.cc:460] Webserver started at http://127.28.62.129:46161/ using document root <none> and password file <none>
I20260809 16:39:33.744772 32527 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:33.744812 32527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:33.744899 32527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:33.745873 32527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "c73b7c630d344065b86a82562e96843e"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "0b8709a95a5b270e17ebb217379e2e2f"
server_key_iv: "38c6a252bd4152de3e72e664ec02c3ad"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.746169 32527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "c73b7c630d344065b86a82562e96843e"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "0b8709a95a5b270e17ebb217379e2e2f"
server_key_iv: "38c6a252bd4152de3e72e664ec02c3ad"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.747484 32527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:33.748164 32546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.748348 32527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:33.748430 32527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "c73b7c630d344065b86a82562e96843e"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "0b8709a95a5b270e17ebb217379e2e2f"
server_key_iv: "38c6a252bd4152de3e72e664ec02c3ad"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.748483 32527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:33.757431 32527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:33.759331 32527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:33.759439 32527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:33.759608 32527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:33.760020 32527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:33.760059 32527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.760080 32527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:33.760125 32527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.765956 32527 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:40781
I20260809 16:39:33.766049 32681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:40781 every 8 connection(s)
I20260809 16:39:33.766355 32527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:33.773576 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 32527
I20260809 16:39:33.773700 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:33.773914 28922 external_mini_cluster.cc:1428] Setting key 21ad238370710d243dc1983d1db40405
I20260809 16:39:33.776849 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:33.767193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:58361 (local address 127.28.62.190:40459)
0809 16:39:33.767288 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:33.767290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:33.767460 (+ 170us) server_negotiation.cc:407] Connection header received
0809 16:39:33.768020 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:33.768021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:33.768034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:33.768088 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:33.769048 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.769575 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.770572 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.770806 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.770895 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:33.773104 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:33.773106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:33.773106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:33.773113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:33.775433 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.775854 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.775855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.775856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.775917 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.776173 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.776174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.776175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.776296 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:33.776369 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:33.776652 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 16:39:33.776758 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 16:39:33.777009 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:33.767100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40459 (local address 127.28.62.129:58361)
0809 16:39:33.767320 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:33.767372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:33.767908 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:33.768163 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:33.768167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:33.768754 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:33.768941 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:33.768945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.769681 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.769682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:33.770476 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:33.770478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.770908 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.770909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:33.771094 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:33.771556 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:33.771564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:33.773017 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:33.775513 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:33.775521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:33.775523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:33.775753 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:33.775994 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:33.775995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:33.775995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:33.776103 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:33.776371 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:33.776379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:33.776512 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 16:39:33.776616 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 16:39:33.776776 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260809 16:39:33.777518 32682 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40459
I20260809 16:39:33.777601 32682 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:33.777801 32682 heartbeater.cc:511] Master 127.28.62.190:40459 requested a full tablet report, sending...
I20260809 16:39:33.778309 32464 ts_manager.cc:194] Registered new tserver with Master: c73b7c630d344065b86a82562e96843e (127.28.62.129:40781)
I20260809 16:39:33.778934 32464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:58361
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:33.802424 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:33.884532 32687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:33.884737 32687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:33.884760 32687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:33.885882 32687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:33.885923 32687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:33.885949 32687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:33.887743 32687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:33.887992 32687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:33.888248 32687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:33.893383 32693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:33.893358 32695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:33.893366 32692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:33.893802 32687 server_base.cc:1034] running on GCE node
I20260809 16:39:33.894014 32687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:33.894264 32687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:33.895428 32687 hybrid_clock.cc:648] HybridClock initialized: now 1786293573895476 us; error 91 us; skew 500 ppm
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:33.897984 32687 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:33.898391 32687 webserver.cc:460] Webserver started at http://127.28.62.130:45949/ using document root <none> and password file <none>
I20260809 16:39:33.898571 32687 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:33.898609 32687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:33.898696 32687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:33.899804 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "c9827af7f42643118adc9fb7ab2afe1b"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "96106dffb8a070518147124280aeef3f"
server_key_iv: "99ad73c90cd638c3c863b73eb705df74"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.900105 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "c9827af7f42643118adc9fb7ab2afe1b"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "96106dffb8a070518147124280aeef3f"
server_key_iv: "99ad73c90cd638c3c863b73eb705df74"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.901412 32687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:33.902109 32705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.902302 32687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:33.902385 32687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "c9827af7f42643118adc9fb7ab2afe1b"
format_stamp: "Formatted at 2026-08-09 16:39:33 on dist-test-slave-wl2h"
server_key: "96106dffb8a070518147124280aeef3f"
server_key_iv: "99ad73c90cd638c3c863b73eb705df74"
server_key_version: "encryptionkey@0"
I20260809 16:39:33.902441 32687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:33.924640 32687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:33.927089 32687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:33.927215 32687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:33.927428 32687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:33.927763 32687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:33.927798 32687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.927821 32687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:33.927872 32687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:33.933980 32687 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:33007
I20260809 16:39:33.934074 369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:33007 every 8 connection(s)
I20260809 16:39:33.934401 32687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:33.938002 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 32687
I20260809 16:39:33.938090 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:33.938303 28922 external_mini_cluster.cc:1428] Setting key bc3a47d5928a5a7bab6d3868aa84c515
Aug 09 16:39:33 dist-test-slave-wl2h krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293573, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:33.944700 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:33.935196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:45157 (local address 127.28.62.190:40459)
0809 16:39:33.935283 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:33.935286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:33.935524 (+ 238us) server_negotiation.cc:407] Connection header received
0809 16:39:33.936499 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:33.936500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:33.936511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:33.936565 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:33.937134 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.937641 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.939181 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.939496 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.939578 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:33.941856 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:33.941857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:33.941858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:33.941865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:33.943252 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.943700 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.943701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.943702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.943766 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:33.944028 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:33.944029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:33.944030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:33.944150 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:33.944229 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:33.944507 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 16:39:33.944594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 16:39:33.944896 372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:33.935153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40459 (local address 127.28.62.130:45157)
0809 16:39:33.935367 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:33.935419 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:33.936409 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:33.936617 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:33.936620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:33.936875 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:33.937063 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:33.937067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.937810 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.937811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:33.939091 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:33.939095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:33.939607 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:33.939608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:33.939718 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:33.940306 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:33.940314 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:33.941757 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:33.943370 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:33.943378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:33.943380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:33.943605 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:33.943836 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:33.943837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:33.943838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:33.943957 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:33.944226 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:33.944234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:33.944374 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 16:39:33.944485 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 16:39:33.944652 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
I20260809 16:39:33.945375 370 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40459
I20260809 16:39:33.945470 370 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:33.945659 370 heartbeater.cc:511] Master 127.28.62.190:40459 requested a full tablet report, sending...
I20260809 16:39:33.946076 32464 ts_manager.cc:194] Registered new tserver with Master: c9827af7f42643118adc9fb7ab2afe1b (127.28.62.130:33007)
I20260809 16:39:33.946467 32464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:45157
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:33.968094 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:34.057436 375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:34.057662 375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:34.057687 375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:34.058979 375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:34.059053 375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:34.059146 375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:34.060781 375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:34.061055 375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:34.061331 375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:34.066591 383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:34.066643 381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:34.066613 380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:34.066768 375 server_base.cc:1034] running on GCE node
I20260809 16:39:34.067070 375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:34.067305 375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:34.068434 375 hybrid_clock.cc:648] HybridClock initialized: now 1786293574068421 us; error 31 us; skew 500 ppm
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293574, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:34.070921 375 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:34.071461 375 webserver.cc:460] Webserver started at http://127.28.62.131:41003/ using document root <none> and password file <none>
I20260809 16:39:34.071733 375 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:34.071784 375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:34.071939 375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:34.073035 375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "ff14a91295794e829304fc1ae46888ea"
format_stamp: "Formatted at 2026-08-09 16:39:34 on dist-test-slave-wl2h"
server_key: "db6d6fb07b5b5cc354dddf4e29982a34"
server_key_iv: "a9efa96992d9bdee82cedc6c031e09e0"
server_key_version: "encryptionkey@0"
I20260809 16:39:34.073405 375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "ff14a91295794e829304fc1ae46888ea"
format_stamp: "Formatted at 2026-08-09 16:39:34 on dist-test-slave-wl2h"
server_key: "db6d6fb07b5b5cc354dddf4e29982a34"
server_key_iv: "a9efa96992d9bdee82cedc6c031e09e0"
server_key_version: "encryptionkey@0"
I20260809 16:39:34.074781 375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:34.075621 392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:34.075829 375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:34.075912 375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "ff14a91295794e829304fc1ae46888ea"
format_stamp: "Formatted at 2026-08-09 16:39:34 on dist-test-slave-wl2h"
server_key: "db6d6fb07b5b5cc354dddf4e29982a34"
server_key_iv: "a9efa96992d9bdee82cedc6c031e09e0"
server_key_version: "encryptionkey@0"
I20260809 16:39:34.075966 375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:34.085775 375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:34.087996 375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:34.088147 375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:34.088446 375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:34.088750 375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:34.088804 375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:34.088843 375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:34.088871 375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:34.094689 375 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44893
I20260809 16:39:34.094786 522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44893 every 8 connection(s)
I20260809 16:39:34.095150 375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293574, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:34.104007 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:34.095891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:42471 (local address 127.28.62.190:40459)
0809 16:39:34.095940 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:34.095943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:34.096165 (+ 222us) server_negotiation.cc:407] Connection header received
0809 16:39:34.096628 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:34.096629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:34.096639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:34.096693 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:34.097560 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.098060 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.099113 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.099317 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.099384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:34.101548 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:34.101549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:34.101549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:34.101556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:34.102831 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:34.103284 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:34.103285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:34.103285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:34.103338 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:34.103533 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:34.103533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:34.103533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:34.103624 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:34.103669 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:34.103876 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 16:39:34.103942 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 16:39:34.104249 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:34.095872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:40459 (local address 127.28.62.131:42471)
0809 16:39:34.096036 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:34.096070 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:34.096550 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:34.096741 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:34.096744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:34.097296 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:34.097477 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:34.097481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.098147 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.098147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:34.098986 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:34.098988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.099406 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.099407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:34.099506 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:34.100035 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:34.100043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:34.101459 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:34.102920 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:34.102926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:34.102928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:34.103212 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:34.103394 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:34.103395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:34.103395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:34.103489 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:34.103674 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:34.103678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:34.103784 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 16:39:34.103863 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 16:39:34.103977 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260809 16:39:34.104333 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 375
I20260809 16:39:34.104415 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:34.104605 28922 external_mini_cluster.cc:1428] Setting key f147459a517176e97ef7f56403b2001e
I20260809 16:39:34.104875 523 heartbeater.cc:347] Connected to a master server at 127.28.62.190:40459
I20260809 16:39:34.104979 523 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:34.105228 523 heartbeater.cc:511] Master 127.28.62.190:40459 requested a full tablet report, sending...
I20260809 16:39:34.105582 32464 ts_manager.cc:194] Registered new tserver with Master: ff14a91295794e829304fc1ae46888ea (127.28.62.131:44893)
I20260809 16:39:34.105921 32464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:42471
I20260809 16:39:34.116457 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[32410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[32410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:34.142012 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:34.133798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.28.62.190:40459)
0809 16:39:34.133971 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:34.133973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:34.134062 (+ 89us) server_negotiation.cc:407] Connection header received
0809 16:39:34.134183 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:34.134184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:34.134197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:34.134274 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:34.134890 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.135484 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.136246 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:34.136442 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:34.136518 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:34.138819 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:34.138820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:34.138820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:34.138827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:34.140733 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:34.141162 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:34.141163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:34.141163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:34.141229 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:34.141433 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:34.141434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:34.141434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:34.141592 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:34.141706 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:34.141815 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:39:34.141898 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
W20260809 16:39:34.142225 32464 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35070
I20260809 16:39:34.142654 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 32527
I20260809 16:39:34.155743 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 32687
I20260809 16:39:34.164494 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 375
I20260809 16:39:34.172514 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 32424
2026-08-09T16:39:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1973 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[536](info): set up 2 sockets
Aug 09 16:39:34 dist-test-slave-wl2h krb5kdc[536](info): commencing operation
krb5kdc: starting...
W20260809 16:39:35.307458 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:35.433702 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45959
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:35.515771 550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:35.515978 550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:35.516000 550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:35.517088 550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:35.517123 550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:35.517138 550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:35.517150 550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:35.517163 550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:35.518705 550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:35.518896 550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:35.519110 550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:35.523941 559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:35.523944 557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:35.524210 550 server_base.cc:1034] running on GCE node
W20260809 16:39:35.523954 556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:35.524546 550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:35.524817 550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:35.525961 550 hybrid_clock.cc:648] HybridClock initialized: now 1786293575525911 us; error 65 us; skew 500 ppm
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:35.528640 550 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:35.529166 550 webserver.cc:460] Webserver started at http://127.28.62.190:38695/ using document root <none> and password file <none>
I20260809 16:39:35.529428 550 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:35.529496 550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:35.529654 550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:35.530725 550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "2ca16b2767384ffbbf28d61140a0cf96"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "2d4b9db46efed756a3470cfbb88323a0"
server_key_iv: "2258ba5b171038ab0d728d1289d74fe4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.531184 550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "2ca16b2767384ffbbf28d61140a0cf96"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "2d4b9db46efed756a3470cfbb88323a0"
server_key_iv: "2258ba5b171038ab0d728d1289d74fe4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.532575 550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:35.533370 566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.533569 550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:35.533671 550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "2ca16b2767384ffbbf28d61140a0cf96"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "2d4b9db46efed756a3470cfbb88323a0"
server_key_iv: "2258ba5b171038ab0d728d1289d74fe4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.533764 550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:35.550166 550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:35.552899 550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:35.553066 550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:35.557904 550 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:45959
I20260809 16:39:35.557931 631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:45959 every 8 connection(s)
I20260809 16:39:35.558461 550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:35.559062 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 550
I20260809 16:39:35.559182 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:35.559227 632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:35.559387 28922 external_mini_cluster.cc:1428] Setting key 0761b79e44d4fd7c896d26d192a9098a
I20260809 16:39:35.562611 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96: Bootstrap starting.
I20260809 16:39:35.563364 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:35.563679 632 log.cc:824] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:35.564468 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96: No bootstrap required, opened a new log
I20260809 16:39:35.566340 632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } }
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:35.566498 632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:35.566548 632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca16b2767384ffbbf28d61140a0cf96, State: Initialized, Role: FOLLOWER
I20260809 16:39:35.566674 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } }
I20260809 16:39:35.566779 632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:35.566820 632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:35.566874 632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:35.567667 632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } }
I20260809 16:39:35.567793 632 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca16b2767384ffbbf28d61140a0cf96; no voters:
I20260809 16:39:35.567974 632 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:35.568037 637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:35.568202 637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca16b2767384ffbbf28d61140a0cf96, State: Running, Role: LEADER
I20260809 16:39:35.568248 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:35.568310 637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } }
I20260809 16:39:35.568614 639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca16b2767384ffbbf28d61140a0cf96. Latest consensus state: current_term: 1 leader_uuid: "2ca16b2767384ffbbf28d61140a0cf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } } }
I20260809 16:39:35.568684 639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:35.568910 643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:35.569410 643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:35.569959 638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca16b2767384ffbbf28d61140a0cf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca16b2767384ffbbf28d61140a0cf96" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45959 } } }
I20260809 16:39:35.570041 638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:35.571625 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:35.559958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.28.62.190:45959)
0809 16:39:35.560163 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:35.560172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:35.560227 (+ 55us) server_negotiation.cc:407] Connection header received
0809 16:39:35.560891 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:35.560899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:35.560963 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:35.561111 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:35.561740 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.563141 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.563972 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.564506 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.564620 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:35.567145 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:35.567149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:35.567149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:35.567161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:35.569291 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.569916 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.569920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.569920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.570034 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.570372 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.570373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.570374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.570693 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:35.570853 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:35.571366 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 16:39:35.571484 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260809 16:39:35.572278 643 catalog_manager.cc:1347] Generated new cluster ID: b8cf25d9e6c8485a902d15fb9ca355b3
I20260809 16:39:35.572338 643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:35.587399 643 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:35.588160 643 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:35.594928 643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca16b2767384ffbbf28d61140a0cf96: Generated new TSK 0
I20260809 16:39:35.595149 643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:35.630956 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:35.713788 663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:35.714048 663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:35.714089 663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:35.715265 663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:35.715337 663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:35.715378 663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:35.716930 663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:35.717229 663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:35.717486 663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:35.722690 668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:35.722738 669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:35.722692 671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:35.722980 663 server_base.cc:1034] running on GCE node
I20260809 16:39:35.723253 663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:35.723467 663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:35.724627 663 hybrid_clock.cc:648] HybridClock initialized: now 1786293575724614 us; error 28 us; skew 500 ppm
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:35.727320 663 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:35.727769 663 webserver.cc:460] Webserver started at http://127.28.62.129:46633/ using document root <none> and password file <none>
I20260809 16:39:35.728018 663 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:35.728065 663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:35.728209 663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:35.729246 663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "b30ae15af4074e5dba195601d9b2f261"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "43b175fa5ee23ef0eae0ff8d4ecb15e9"
server_key_iv: "41d2f9bf90615140406eb2598106fe8b"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.729594 663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "b30ae15af4074e5dba195601d9b2f261"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "43b175fa5ee23ef0eae0ff8d4ecb15e9"
server_key_iv: "41d2f9bf90615140406eb2598106fe8b"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.730962 663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:35.731751 679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.731954 663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:35.732055 663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "b30ae15af4074e5dba195601d9b2f261"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "43b175fa5ee23ef0eae0ff8d4ecb15e9"
server_key_iv: "41d2f9bf90615140406eb2598106fe8b"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.732111 663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:35.756445 663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:35.758548 663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:35.758672 663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:35.758900 663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:35.759294 663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:35.759332 663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.759356 663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:35.759408 663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.765440 663 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:36061
I20260809 16:39:35.765496 811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:36061 every 8 connection(s)
I20260809 16:39:35.765815 663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:35.767057 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 663
I20260809 16:39:35.767201 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:35.767416 28922 external_mini_cluster.cc:1428] Setting key 699b5fd074c814dac0cad5a764e13fc3
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:35.777168 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:35.766558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:52011 (local address 127.28.62.190:45959)
0809 16:39:35.766645 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:35.766647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:35.766805 (+ 158us) server_negotiation.cc:407] Connection header received
0809 16:39:35.767996 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:35.767998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:35.768011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:35.768065 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:35.768839 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.769428 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.770839 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.771178 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.771252 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:35.773961 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:35.773962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:35.773963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:35.773971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:35.775705 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.776136 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.776137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.776138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.776202 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.776455 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.776456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.776457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.776606 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:35.776694 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:35.776958 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 16:39:35.777054 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 16:39:35.777319 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:35.766471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45959 (local address 127.28.62.129:52011)
0809 16:39:35.766648 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:35.766691 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:35.767888 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:35.768139 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:35.768143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:35.768539 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:35.768744 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:35.768748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.769517 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.769518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:35.770736 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:35.770739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.771275 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.771276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:35.771418 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:35.771948 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:35.771958 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:35.773856 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:35.775787 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:35.775794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:35.775796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:35.776050 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:35.776278 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:35.776279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:35.776279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:35.776385 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:35.776688 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:35.776693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:35.776813 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 16:39:35.776903 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:39:35.777033 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:39:35.778043 812 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45959
I20260809 16:39:35.778149 812 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:35.778292 812 heartbeater.cc:511] Master 127.28.62.190:45959 requested a full tablet report, sending...
I20260809 16:39:35.778846 588 ts_manager.cc:194] Registered new tserver with Master: b30ae15af4074e5dba195601d9b2f261 (127.28.62.129:36061)
I20260809 16:39:35.779484 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:52011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:35.799275 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:35.881793 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:35.881989 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:35.882012 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:35.883085 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:35.883119 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:35.883145 817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:35.884644 817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:35.884832 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:35.885007 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:35.890163 823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:35.890179 824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:35.890166 826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:35.890389 817 server_base.cc:1034] running on GCE node
I20260809 16:39:35.890650 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:35.890887 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:35.892058 817 hybrid_clock.cc:648] HybridClock initialized: now 1786293575892031 us; error 30 us; skew 500 ppm
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:35.894719 817 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:35.895258 817 webserver.cc:460] Webserver started at http://127.28.62.130:33163/ using document root <none> and password file <none>
I20260809 16:39:35.895511 817 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:35.895581 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:35.895759 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:35.896829 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "b3ba5edbc3554e8a8a2b57cb6b14cebf"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "e4ed5b109db72536413070dbb217c5b5"
server_key_iv: "4f0a172563d9a247621e6b3c599d68d4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.897131 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "b3ba5edbc3554e8a8a2b57cb6b14cebf"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "e4ed5b109db72536413070dbb217c5b5"
server_key_iv: "4f0a172563d9a247621e6b3c599d68d4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.898370 817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:35.899075 833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.899255 817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:35.899344 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b3ba5edbc3554e8a8a2b57cb6b14cebf"
format_stamp: "Formatted at 2026-08-09 16:39:35 on dist-test-slave-wl2h"
server_key: "e4ed5b109db72536413070dbb217c5b5"
server_key_iv: "4f0a172563d9a247621e6b3c599d68d4"
server_key_version: "encryptionkey@0"
I20260809 16:39:35.899400 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:35.916101 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:35.918404 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:35.918522 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:35.918726 817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:35.919063 817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:35.919112 817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.919140 817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:35.919154 817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:35.925261 817 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36237
I20260809 16:39:35.925338 969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36237 every 8 connection(s)
I20260809 16:39:35.925678 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:39:35 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:35.934732 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 817
I20260809 16:39:35.934867 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:35.935117 28922 external_mini_cluster.cc:1428] Setting key cec7713ab79d0f1c6b1a5af1983def9f
I20260809 16:39:35.935329 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:35.926490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:38725 (local address 127.28.62.190:45959)
0809 16:39:35.926578 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:35.926580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:35.926765 (+ 185us) server_negotiation.cc:407] Connection header received
0809 16:39:35.927827 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:35.927829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:35.927847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:35.927904 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:35.928534 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.929031 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.930047 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.930249 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.930307 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:35.932521 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:35.932522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:35.932522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:35.932529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:35.933743 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.934102 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.934103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.934103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.934152 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:35.934444 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:35.934445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:35.934445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:35.934616 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:35.934687 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:35.935091 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 16:39:35.935175 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 16:39:35.935534 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:35.926441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45959 (local address 127.28.62.130:38725)
0809 16:39:35.926638 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:35.926678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:35.927713 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:35.927978 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:35.927981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:35.928265 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:35.928451 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:35.928455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.929109 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.929110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:35.929969 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:35.929972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:35.930330 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:35.930330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:35.930427 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:35.930954 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:35.930962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:35.932428 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:35.933815 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:35.933821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:35.933822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:35.934037 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:35.934243 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:35.934243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:35.934244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:35.934354 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:35.934705 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:35.934712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:35.934905 (+ 193us) client_negotiation.cc:769] Sending connection context
0809 16:39:35.935054 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 16:39:35.935240 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260809 16:39:35.936117 970 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45959
I20260809 16:39:35.936251 970 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:35.936427 970 heartbeater.cc:511] Master 127.28.62.190:45959 requested a full tablet report, sending...
I20260809 16:39:35.936904 588 ts_manager.cc:194] Registered new tserver with Master: b3ba5edbc3554e8a8a2b57cb6b14cebf (127.28.62.130:36237)
I20260809 16:39:35.937289 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:38725
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:35.962291 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:36.045306 975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:36.045507 975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:36.045531 975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:36.046669 975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:36.046718 975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:36.046746 975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:36.048336 975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:36.048597 975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:36.048839 975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:36.053872 984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:36.053887 982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:36.053936 975 server_base.cc:1034] running on GCE node
W20260809 16:39:36.053882 981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:36.054353 975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:36.054595 975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:36.055791 975 hybrid_clock.cc:648] HybridClock initialized: now 1786293576055772 us; error 31 us; skew 500 ppm
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:36.058784 975 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:36.059331 975 webserver.cc:460] Webserver started at http://127.28.62.131:41577/ using document root <none> and password file <none>
I20260809 16:39:36.059576 975 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:36.059656 975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:36.059808 975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:36.060861 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "886529bc3f4c4d99a04ac345606f6182"
format_stamp: "Formatted at 2026-08-09 16:39:36 on dist-test-slave-wl2h"
server_key: "fea9bb2af039dbd2d66c67c22357f777"
server_key_iv: "f5cff2849ba0ddcb8639341600a11bb0"
server_key_version: "encryptionkey@0"
I20260809 16:39:36.061244 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "886529bc3f4c4d99a04ac345606f6182"
format_stamp: "Formatted at 2026-08-09 16:39:36 on dist-test-slave-wl2h"
server_key: "fea9bb2af039dbd2d66c67c22357f777"
server_key_iv: "f5cff2849ba0ddcb8639341600a11bb0"
server_key_version: "encryptionkey@0"
I20260809 16:39:36.062602 975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:36.063409 993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:36.063591 975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:36.063699 975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "886529bc3f4c4d99a04ac345606f6182"
format_stamp: "Formatted at 2026-08-09 16:39:36 on dist-test-slave-wl2h"
server_key: "fea9bb2af039dbd2d66c67c22357f777"
server_key_iv: "f5cff2849ba0ddcb8639341600a11bb0"
server_key_version: "encryptionkey@0"
I20260809 16:39:36.063776 975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:36.074450 975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:36.076818 975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:36.076980 975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:36.077214 975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:36.077524 975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:36.077580 975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:36.077621 975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:36.077651 975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:36.083670 975 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:42769
I20260809 16:39:36.083719 1124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:42769 every 8 connection(s)
I20260809 16:39:36.084018 975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:36.088107 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 975
I20260809 16:39:36.088205 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:36.088382 28922 external_mini_cluster.cc:1428] Setting key d4839100da13f1f8fc464de8097ddd5d
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:36.094007 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:36.084740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:55089 (local address 127.28.62.190:45959)
0809 16:39:36.084805 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:36.084811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:36.085028 (+ 217us) server_negotiation.cc:407] Connection header received
0809 16:39:36.085929 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:36.085931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:36.085941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:36.086008 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:36.086682 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.087365 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.088655 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.088972 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.089041 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:36.091343 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:36.091345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:36.091345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:36.091351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:36.092836 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.093225 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.093225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.093226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.093274 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.093498 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.093498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.093498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.093594 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:36.093642 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:36.093875 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 16:39:36.093944 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 16:39:36.094208 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:36.084713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45959 (local address 127.28.62.131:55089)
0809 16:39:36.084874 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:36.084916 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:36.085816 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:36.086058 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:36.086061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:36.086344 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:36.086586 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:36.086591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.087455 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.087456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:36.088552 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:36.088555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.089107 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.089108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:36.089255 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:36.089736 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:36.089743 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:36.091255 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:36.092914 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:36.092921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:36.092923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:36.093140 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:36.093335 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:36.093335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:36.093336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:36.093441 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:36.093649 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:36.093652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:36.093760 (+ 108us) client_negotiation.cc:769] Sending connection context
0809 16:39:36.093846 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:36.093965 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260809 16:39:36.094734 1125 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45959
I20260809 16:39:36.094846 1125 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:36.095074 1125 heartbeater.cc:511] Master 127.28.62.190:45959 requested a full tablet report, sending...
I20260809 16:39:36.095475 588 ts_manager.cc:194] Registered new tserver with Master: 886529bc3f4c4d99a04ac345606f6182 (127.28.62.131:42769)
I20260809 16:39:36.095846 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:55089
I20260809 16:39:36.100073 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:36.124315 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:36.116456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.28.62.129:36061)
0809 16:39:36.116613 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:36.116638 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:36.116649 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:36.116677 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:36.116678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:36.116692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:36.116767 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:36.117311 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.118026 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.118794 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.119202 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.119263 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:36.121552 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:36.121561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:36.121561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:36.121568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:36.123259 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.123585 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.123587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.123588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.123683 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.123859 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.123860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.123860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.123983 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:36.124092 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:36.124172 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:39:36.124233 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:36.132694 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:36.125736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.28.62.190:45959)
0809 16:39:36.125796 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:36.125798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:36.125806 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:36.125836 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:36.125837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:36.125846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:36.125891 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:36.126381 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.126876 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.127827 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:36.128014 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:36.128068 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:36.130261 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:36.130262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:36.130262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:36.130269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:36.131664 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.131964 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.131964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.131964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.132009 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:36.132242 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:36.132243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:36.132243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:36.132379 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:36.132457 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:36.132546 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 16:39:36.132602 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 16:39:36.132992 588 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42258
I20260809 16:39:36.133435 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 663
I20260809 16:39:36.141248 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 817
I20260809 16:39:36.146673 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 975
I20260809 16:39:36.152817 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 550
2026-08-09T16:39:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1979 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[1140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[1140](info): set up 2 sockets
Aug 09 16:39:36 dist-test-slave-wl2h krb5kdc[1140](info): commencing operation
krb5kdc: starting...
W20260809 16:39:37.293078 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:37.417677 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36513
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:37.499714 1154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:37.499912 1154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:37.499934 1154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:37.501060 1154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:37.501109 1154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:37.501124 1154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:37.501137 1154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:37.501190 1154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:37.502642 1154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:37.502878 1154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:37.503152 1154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:37.507807 1160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.507805 1163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.507830 1161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:37.508181 1154 server_base.cc:1034] running on GCE node
I20260809 16:39:37.508421 1154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:37.508713 1154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:37.509858 1154 hybrid_clock.cc:648] HybridClock initialized: now 1786293577509835 us; error 47 us; skew 500 ppm
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:37.516278 1154 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:37.516659 1154 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:37.516737 1154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:37.516906 1154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:37.518044 1154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "88cba194f0de4554b79c610ae5b750db"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "fcf1f9422f62ba568114333a0b9079c1"
server_key_iv: "fe641a01671bb88f38c69b5b200acfd1"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.518460 1154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "88cba194f0de4554b79c610ae5b750db"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "fcf1f9422f62ba568114333a0b9079c1"
server_key_iv: "fe641a01671bb88f38c69b5b200acfd1"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.519982 1154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:37.520777 1169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.520964 1154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:37.521082 1154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "88cba194f0de4554b79c610ae5b750db"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "fcf1f9422f62ba568114333a0b9079c1"
server_key_iv: "fe641a01671bb88f38c69b5b200acfd1"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.521176 1154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:37.533469 1154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:37.535858 1154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:37.536037 1154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:37.540501 1154 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:36513
I20260809 16:39:37.540529 1226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:36513 every 8 connection(s)
I20260809 16:39:37.540844 1154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:37.541785 1227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:37.542721 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1154
I20260809 16:39:37.542809 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:37.543059 28922 external_mini_cluster.cc:1428] Setting key d6dbd3680548907cab3e191021ba53eb
I20260809 16:39:37.544540 1227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: Bootstrap starting.
I20260809 16:39:37.545367 1227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:37.545723 1227 log.cc:824] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:37.546437 1227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: No bootstrap required, opened a new log
I20260809 16:39:37.548403 1227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } }
I20260809 16:39:37.548574 1227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:37.548625 1227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cba194f0de4554b79c610ae5b750db, State: Initialized, Role: FOLLOWER
I20260809 16:39:37.548795 1227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } }
I20260809 16:39:37.548892 1227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:37.548936 1227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:37.548985 1227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:37.549520 1227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } }
I20260809 16:39:37.549635 1227 leader_election.cc:302] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88cba194f0de4554b79c610ae5b750db; no voters:
I20260809 16:39:37.549809 1227 leader_election.cc:288] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:37.549837 1235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:37.549939 1235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [term 1 LEADER]: Becoming Leader. State: Replica: 88cba194f0de4554b79c610ae5b750db, State: Running, Role: LEADER
I20260809 16:39:37.550036 1235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } }
I20260809 16:39:37.550063 1227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:37.551831 1235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88cba194f0de4554b79c610ae5b750db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } } }
I20260809 16:39:37.551918 1235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:37.552028 1235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88cba194f0de4554b79c610ae5b750db. Latest consensus state: current_term: 1 leader_uuid: "88cba194f0de4554b79c610ae5b750db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cba194f0de4554b79c610ae5b750db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36513 } } }
I20260809 16:39:37.552101 1235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db [sys.catalog]: This master's current role is: LEADER
W20260809 16:39:37.552968 1253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:39:37.553023 1253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:39:37.553069 1252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:37.553231 1252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:37.554782 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:37.543675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.28.62.190:36513)
0809 16:39:37.543845 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:37.543853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:37.543879 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:39:37.544416 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:37.544423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:37.544469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:37.544593 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:37.545460 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.546489 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.547333 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.547750 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.547872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:37.550984 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:37.550988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:37.550988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:37.550999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:37.553174 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.553534 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.553537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.553538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.553653 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.553895 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.553896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.553896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.554183 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:37.554370 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:37.554439 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 16:39:37.554575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260809 16:39:37.554924 1252 catalog_manager.cc:1347] Generated new cluster ID: c766132ce1a441fd993f243e5410981d
I20260809 16:39:37.554966 1252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:37.560814 1252 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:37.561333 1252 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:37.568728 1252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: Generated new TSK 0
I20260809 16:39:37.568926 1252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:37.598026 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:37.686829 1261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:37.687112 1261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:37.687144 1261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:37.688320 1261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:37.688364 1261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:37.688392 1261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:37.689883 1261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:37.690132 1261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:37.690390 1261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:37.697155 1271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.697149 1267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.697173 1268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:37.697438 1261 server_base.cc:1034] running on GCE node
I20260809 16:39:37.697678 1261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:37.697965 1261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:37.699075 1261 hybrid_clock.cc:648] HybridClock initialized: now 1786293577699061 us; error 29 us; skew 500 ppm
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:37.702257 1261 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:37.702582 1261 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:37.702657 1261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:37.702807 1261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:37.703925 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "84b5ce07e3ef4a6ab035b464b4042826"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "7f871e9f56bc91485dbdb8440a526609"
server_key_iv: "e30d39c842f21f061ecee6d4a5b40372"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.704361 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "84b5ce07e3ef4a6ab035b464b4042826"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "7f871e9f56bc91485dbdb8440a526609"
server_key_iv: "e30d39c842f21f061ecee6d4a5b40372"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.706220 1261 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:37.707052 1278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.707265 1261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:37.707396 1261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "84b5ce07e3ef4a6ab035b464b4042826"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "7f871e9f56bc91485dbdb8440a526609"
server_key_iv: "e30d39c842f21f061ecee6d4a5b40372"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.707486 1261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:37.714751 1261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:37.717151 1261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:37.717319 1261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:37.717563 1261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:37.717891 1261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:37.717947 1261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.717991 1261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:37.718015 1261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.724754 1261 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:35239
I20260809 16:39:37.724807 1403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:35239 every 8 connection(s)
I20260809 16:39:37.725111 1261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:37.735061 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1261
I20260809 16:39:37.735199 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:37.735414 28922 external_mini_cluster.cc:1428] Setting key 55ad34b57c96bb627797926e20784c23
I20260809 16:39:37.736125 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:37.725903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:43205 (local address 127.28.62.190:36513)
0809 16:39:37.725994 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:37.725997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:37.726137 (+ 140us) server_negotiation.cc:407] Connection header received
0809 16:39:37.726646 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:37.726647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:37.726661 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:37.726753 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:37.727755 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.728696 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.729766 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.730147 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.730236 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:37.732594 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:37.732596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:37.732596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:37.732606 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:37.734441 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.734893 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.734895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.734895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.734962 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.735368 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.735369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.735369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.735538 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:37.735618 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:37.735936 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 16:39:37.736034 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 16:39:37.736496 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:37.725832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:36513 (local address 127.28.62.129:43205)
0809 16:39:37.726019 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:37.726063 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:37.726551 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:37.726803 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:37.726806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:37.727169 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:37.727660 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:37.727665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.728824 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.728826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:37.729672 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:37.729675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.730243 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.730245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:37.730351 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:37.730948 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:37.730956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:37.732499 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:37.734546 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:37.734552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:37.734554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:37.734783 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:37.735069 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:37.735070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:37.735070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:37.735289 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:37.735621 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:37.735629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:37.735788 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 16:39:37.735913 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 16:39:37.736085 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260809 16:39:37.738278 1404 heartbeater.cc:347] Connected to a master server at 127.28.62.190:36513
I20260809 16:39:37.738400 1404 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:37.738569 1404 heartbeater.cc:511] Master 127.28.62.190:36513 requested a full tablet report, sending...
I20260809 16:39:37.739177 1187 ts_manager.cc:194] Registered new tserver with Master: 84b5ce07e3ef4a6ab035b464b4042826 (127.28.62.129:35239)
I20260809 16:39:37.739822 1187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:43205
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:37.770179 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:37.853854 1414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:37.854048 1414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:37.854071 1414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:37.855211 1414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:37.855248 1414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:37.855275 1414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:37.856694 1414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:37.856884 1414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:37.857048 1414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:37.861919 1422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.861920 1424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:37.861919 1421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:37.862290 1414 server_base.cc:1034] running on GCE node
I20260809 16:39:37.862514 1414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:37.862761 1414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:37.863895 1414 hybrid_clock.cc:648] HybridClock initialized: now 1786293577863883 us; error 27 us; skew 500 ppm
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:37.866353 1414 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:37.866698 1414 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:37.866776 1414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:37.866925 1414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:37.867969 1414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "6aa1d58dfaf4a59fe27114666ded94bf"
server_key_iv: "affef157984c70b2ebaf4a4033522643"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.868335 1414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "6aa1d58dfaf4a59fe27114666ded94bf"
server_key_iv: "affef157984c70b2ebaf4a4033522643"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.869642 1414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:37.870393 1430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.870558 1414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:37.870657 1414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd"
format_stamp: "Formatted at 2026-08-09 16:39:37 on dist-test-slave-wl2h"
server_key: "6aa1d58dfaf4a59fe27114666ded94bf"
server_key_iv: "affef157984c70b2ebaf4a4033522643"
server_key_version: "encryptionkey@0"
I20260809 16:39:37.870743 1414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:37.880839 1414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:37.882771 1414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:37.882929 1414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:37.883222 1414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:37.883546 1414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:37.883602 1414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.883638 1414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:37.883687 1414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:37.889603 1414 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:43259
I20260809 16:39:37.889647 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:43259 every 8 connection(s)
I20260809 16:39:37.889906 1414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:37.895561 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1414
I20260809 16:39:37.895671 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:37.895880 28922 external_mini_cluster.cc:1428] Setting key 408bffa7d0de8fb5c85b3e4c47c7be95
Aug 09 16:39:37 dist-test-slave-wl2h krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:37.899262 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:37.890598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:50247 (local address 127.28.62.190:36513)
0809 16:39:37.890668 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:37.890670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:37.890886 (+ 216us) server_negotiation.cc:407] Connection header received
0809 16:39:37.891394 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:37.891395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:37.891405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:37.891462 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:37.892369 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.892870 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.893841 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.894052 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.894154 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:37.896528 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:37.896529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:37.896529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:37.896535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:37.897840 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.898264 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.898266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.898266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.898328 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:37.898594 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:37.898594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:37.898595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:37.898715 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:37.898787 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:37.899083 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 16:39:37.899172 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 16:39:37.899461 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:37.890567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:36513 (local address 127.28.62.130:50247)
0809 16:39:37.890757 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:37.890798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:37.891308 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:37.891511 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:37.891514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:37.891780 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:37.892277 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:37.892282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.892950 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.892951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:37.893733 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:37.893735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:37.894167 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:37.894168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:37.894302 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:37.894758 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:37.894766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:37.896412 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:37.897942 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:37.897950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:37.897952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:37.898168 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:37.898396 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:37.898397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:37.898397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:37.898520 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:37.898790 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:37.898797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:37.898930 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 16:39:37.899067 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 16:39:37.899223 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260809 16:39:37.899976 1562 heartbeater.cc:347] Connected to a master server at 127.28.62.190:36513
I20260809 16:39:37.900076 1562 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:37.900234 1562 heartbeater.cc:511] Master 127.28.62.190:36513 requested a full tablet report, sending...
I20260809 16:39:37.900601 1187 ts_manager.cc:194] Registered new tserver with Master: 3bea39b269ef4dc8a8819eef4ff0c5fd (127.28.62.130:43259)
I20260809 16:39:37.901048 1187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:50247
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:37.922621 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:38.004920 1568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.005162 1568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.005214 1568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.006374 1568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.006444 1568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.006487 1568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:38.008013 1568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.008291 1568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.008525 1568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.013599 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.013676 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.013662 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.013808 1568 server_base.cc:1034] running on GCE node
I20260809 16:39:38.014248 1568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.014495 1568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.015669 1568 hybrid_clock.cc:648] HybridClock initialized: now 1786293578015645 us; error 56 us; skew 500 ppm
Aug 09 16:39:38 dist-test-slave-wl2h krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293578, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:38.018431 1568 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:38.018819 1568 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.018894 1568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.019105 1568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.020179 1568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "ee844f0b498a43378010588db106096c"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "6b0598982bee256adf1b59d5b4b65eba"
server_key_iv: "f8ceaa57c4d926361f6a2016a23f910d"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.020545 1568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "ee844f0b498a43378010588db106096c"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "6b0598982bee256adf1b59d5b4b65eba"
server_key_iv: "f8ceaa57c4d926361f6a2016a23f910d"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.021926 1568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:38.022717 1586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.022887 1568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:38.022996 1568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "ee844f0b498a43378010588db106096c"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "6b0598982bee256adf1b59d5b4b65eba"
server_key_iv: "f8ceaa57c4d926361f6a2016a23f910d"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.023124 1568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.043560 1568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.045747 1568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.045934 1568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.046204 1568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:38.046568 1568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:38.046603 1568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.046627 1568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:38.046640 1568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.053277 1568 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:34797
I20260809 16:39:38.053325 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:34797 every 8 connection(s)
I20260809 16:39:38.053618 1568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:38.058063 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1568
I20260809 16:39:38.058162 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:38.058364 28922 external_mini_cluster.cc:1428] Setting key 412fb2b201c40f40f53173ff9e9c7490
Aug 09 16:39:38 dist-test-slave-wl2h krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293578, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:38.062507 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.054334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:57669 (local address 127.28.62.190:36513)
0809 16:39:38.054419 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.054422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.054599 (+ 177us) server_negotiation.cc:407] Connection header received
0809 16:39:38.055124 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.055125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.055135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.055178 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.056042 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.056539 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.057531 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.057716 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.057765 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.059927 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.059928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.059929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:38.059935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.061423 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:38.061770 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:38.061771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:38.061771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:38.061820 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:38.062023 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:38.062023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:38.062024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:38.062118 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.062167 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.062383 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.062446 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 16:39:38.062722 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.054297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:36513 (local address 127.28.62.131:57669)
0809 16:39:38.054476 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.054513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.054992 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.055232 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.055235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.055494 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.055965 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.055969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.056623 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.056623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.057455 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.057457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.057788 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.057789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.057885 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.058466 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:38.058476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.059850 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.061496 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:38.061501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:38.061502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:38.061705 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:38.061878 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:38.061879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:38.061879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:38.061969 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:38.062172 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.062176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.062279 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.062357 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.062475 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260809 16:39:38.063230 1714 heartbeater.cc:347] Connected to a master server at 127.28.62.190:36513
I20260809 16:39:38.063311 1714 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:38.063421 1714 heartbeater.cc:511] Master 127.28.62.190:36513 requested a full tablet report, sending...
I20260809 16:39:38.063751 1187 ts_manager.cc:194] Registered new tserver with Master: ee844f0b498a43378010588db106096c (127.28.62.131:34797)
I20260809 16:39:38.064116 1187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:57669
I20260809 16:39:38.069828 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:38.076671 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.071153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.28.62.190:36513)
0809 16:39:38.071208 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.071210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.071300 (+ 90us) server_negotiation.cc:407] Connection header received
0809 16:39:38.071336 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.071337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.071345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.071399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.071894 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.072363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.073014 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.073187 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.073241 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.074436 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.074437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.074437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:38.074442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.075733 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:38.076063 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:38.076064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:38.076064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:38.076112 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:38.076268 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:38.076269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:38.076269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:38.076397 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.076471 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.076560 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.076615 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:38.078289 1187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:38.079878 1187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:38.086158 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.081979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:34797 (local address 127.0.0.1:50602)
0809 16:39:38.082231 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.082272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.082350 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.082623 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.082626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.082628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:38.082691 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.082693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.084109 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.084109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.085133 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.085135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.085846 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.085930 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.085943 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.086032 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.086085 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":3,"thread_start_us":47,"threads_started":1}
I20260809 16:39:38.086175 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.082093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.28.62.131:34797)
0809 16:39:38.082268 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.082272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.082327 (+ 55us) server_negotiation.cc:407] Connection header received
0809 16:39:38.082480 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.082483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.082501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.082614 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:38.082900 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.084004 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085211 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.085746 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085831 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.085887 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.086054 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.086095 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260809 16:39:38.086373 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.082292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35239 (local address 127.0.0.1:50316)
0809 16:39:38.082410 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.082419 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.082462 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.083082 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.083083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.083086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:38.083126 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.083127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.084336 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.084337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.085280 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.085282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.086119 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.086120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.086186 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.086189 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.086250 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.086299 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":83,"threads_started":1}
I20260809 16:39:38.086596 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.081984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:43259 (local address 127.0.0.1:34034)
0809 16:39:38.082314 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.082325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.082382 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.082718 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.082719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.082721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:38.082756 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.082758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.083746 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.083747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.084934 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.084937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.086356 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.086357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.086419 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.086423 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.086476 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.086520 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":188,"threads_started":1}
I20260809 16:39:38.086601 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.082560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.28.62.129:35239)
0809 16:39:38.082832 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.082837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.082846 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:38.082885 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.082887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.082901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.082976 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:38.083219 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.084234 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085364 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.086032 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.086118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.086169 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.086464 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.086514 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260809 16:39:38.086822 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.082149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.28.62.130:43259)
0809 16:39:38.082497 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.082501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.082510 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:38.082545 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.082547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.082561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.082634 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:38.082815 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.083625 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085029 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.085741 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.085840 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.085891 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.086707 (+ 816us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.086750 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":81,"threads_started":1}
I20260809 16:39:38.086927 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4ef824f14541b1b968c939e060f20e (DEFAULT_TABLE table=test-table [id=ade2802e412f475da98d6b2c7afd0ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.087224 1333 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4ef824f14541b1b968c939e060f20e (DEFAULT_TABLE table=test-table [id=ade2802e412f475da98d6b2c7afd0ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.087380 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4ef824f14541b1b968c939e060f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.087545 1333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4ef824f14541b1b968c939e060f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.089885 1736 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Bootstrap starting.
I20260809 16:39:38.090131 1737 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: Bootstrap starting.
I20260809 16:39:38.090655 1484 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4ef824f14541b1b968c939e060f20e (DEFAULT_TABLE table=test-table [id=ade2802e412f475da98d6b2c7afd0ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.090863 1736 tablet_bootstrap.cc:654] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.090974 1484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4ef824f14541b1b968c939e060f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.091521 1736 log.cc:824] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.092309 1736 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: No bootstrap required, opened a new log
I20260809 16:39:38.092392 1736 ts_tablet_manager.cc:1397] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:39:38.092509 1737 tablet_bootstrap.cc:654] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.093151 1737 log.cc:824] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.093948 1737 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: No bootstrap required, opened a new log
I20260809 16:39:38.094029 1737 ts_tablet_manager.cc:1397] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 16:39:38.094578 1740 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Bootstrap starting.
I20260809 16:39:38.094570 1736 raft_consensus.cc:348] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.094740 1736 raft_consensus.cc:374] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.094786 1736 raft_consensus.cc:729] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee844f0b498a43378010588db106096c, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.094919 1736 consensus_queue.cc:260] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.095224 1736 ts_tablet_manager.cc:1428] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:38.095254 1740 tablet_bootstrap.cc:654] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.095360 1714 heartbeater.cc:503] Master 127.28.62.190:36513 was elected leader, sending a full tablet report...
I20260809 16:39:38.095911 1740 log.cc:824] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.096397 1737 raft_consensus.cc:348] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.096556 1737 raft_consensus.cc:374] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.096609 1737 raft_consensus.cc:729] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b5ce07e3ef4a6ab035b464b4042826, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.096726 1740 tablet_bootstrap.cc:492] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: No bootstrap required, opened a new log
I20260809 16:39:38.096725 1737 consensus_queue.cc:260] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.096793 1740 ts_tablet_manager.cc:1397] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:38.096997 1737 ts_tablet_manager.cc:1428] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:38.097028 1404 heartbeater.cc:503] Master 127.28.62.190:36513 was elected leader, sending a full tablet report...
I20260809 16:39:38.098729 1740 raft_consensus.cc:348] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.098848 1740 raft_consensus.cc:374] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.098870 1740 raft_consensus.cc:729] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bea39b269ef4dc8a8819eef4ff0c5fd, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.098973 1740 consensus_queue.cc:260] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.099272 1740 ts_tablet_manager.cc:1428] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:38.099308 1562 heartbeater.cc:503] Master 127.28.62.190:36513 was elected leader, sending a full tablet report...
I20260809 16:39:38.111260 1744 raft_consensus.cc:482] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:38.111337 1744 raft_consensus.cc:504] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.111593 1744 leader_election.cc:288] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b5ce07e3ef4a6ab035b464b4042826 (127.28.62.129:35239), ee844f0b498a43378010588db106096c (127.28.62.131:34797)
I20260809 16:39:38.114423 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.111748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:34797 (local address 127.28.62.130:35561)
0809 16:39:38.111813 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.111825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.111866 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.111999 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.112000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.112002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:38.112081 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.112083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.112827 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.112828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.113620 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.113623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.114170 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.114171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.114254 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.114257 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.114305 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.114335 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 16:39:38.114437 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.111828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:35561 (local address 127.28.62.131:34797)
0809 16:39:38.111873 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.111875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.111887 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:38.111916 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.111916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.111926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.111980 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:38.112167 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.112734 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.113714 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.114094 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.114147 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.114154 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.114329 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.114374 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:38.114509 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.111828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:36239 (local address 127.28.62.129:35239)
0809 16:39:38.111912 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.111914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.112014 (+ 100us) server_negotiation.cc:407] Connection header received
0809 16:39:38.112162 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.112163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.112172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.112221 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:38.112364 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.112828 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.113742 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.114147 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.114220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.114229 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.114398 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.114436 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 16:39:38.114718 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.111748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35239 (local address 127.28.62.130:36239)
0809 16:39:38.111963 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.111972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.112036 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.112238 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.112239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.112241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:38.112302 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.112303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.112908 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.112909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.113632 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:38.113633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.114236 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.114237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:38.114319 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.114324 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.114381 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.114416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260809 16:39:38.114801 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4ef824f14541b1b968c939e060f20e" candidate_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee844f0b498a43378010588db106096c" is_pre_election: true
I20260809 16:39:38.114939 1662 raft_consensus.cc:2454] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bea39b269ef4dc8a8819eef4ff0c5fd in term 0.
I20260809 16:39:38.115068 1355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4ef824f14541b1b968c939e060f20e" candidate_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b5ce07e3ef4a6ab035b464b4042826" is_pre_election: true
I20260809 16:39:38.115170 1435 leader_election.cc:302] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bea39b269ef4dc8a8819eef4ff0c5fd, ee844f0b498a43378010588db106096c; no voters:
I20260809 16:39:38.115221 1355 raft_consensus.cc:2454] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bea39b269ef4dc8a8819eef4ff0c5fd in term 0.
I20260809 16:39:38.115284 1744 raft_consensus.cc:2781] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:38.115352 1744 raft_consensus.cc:482] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:38.115388 1744 raft_consensus.cc:3037] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.116000 1744 raft_consensus.cc:504] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.116182 1744 leader_election.cc:288] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [CANDIDATE]: Term 1 election: Requested vote from peers 84b5ce07e3ef4a6ab035b464b4042826 (127.28.62.129:35239), ee844f0b498a43378010588db106096c (127.28.62.131:34797)
I20260809 16:39:38.116326 1355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4ef824f14541b1b968c939e060f20e" candidate_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b5ce07e3ef4a6ab035b464b4042826"
I20260809 16:39:38.116343 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4ef824f14541b1b968c939e060f20e" candidate_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee844f0b498a43378010588db106096c"
I20260809 16:39:38.116401 1355 raft_consensus.cc:3037] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.116401 1662 raft_consensus.cc:3037] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.117242 1662 raft_consensus.cc:2454] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bea39b269ef4dc8a8819eef4ff0c5fd in term 1.
I20260809 16:39:38.117242 1355 raft_consensus.cc:2454] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bea39b269ef4dc8a8819eef4ff0c5fd in term 1.
I20260809 16:39:38.117416 1435 leader_election.cc:302] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bea39b269ef4dc8a8819eef4ff0c5fd, ee844f0b498a43378010588db106096c; no voters:
I20260809 16:39:38.117537 1744 raft_consensus.cc:2781] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:38.117705 1744 raft_consensus.cc:686] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 1 LEADER]: Becoming Leader. State: Replica: 3bea39b269ef4dc8a8819eef4ff0c5fd, State: Running, Role: LEADER
I20260809 16:39:38.117810 1744 consensus_queue.cc:237] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } }
I20260809 16:39:38.118513 1185 catalog_manager.cc:5495] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bea39b269ef4dc8a8819eef4ff0c5fd (127.28.62.130). New cstate: current_term: 1 leader_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bea39b269ef4dc8a8819eef4ff0c5fd" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:38.123574 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.121462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.28.62.130:43259)
0809 16:39:38.121502 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.121503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.121510 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:38.121555 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.121555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.121569 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.121620 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:38.121722 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.122130 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.122847 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.123057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.123121 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.123317 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:38.123399 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:38.123457 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.123498 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.123514 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:38.125216 1355 raft_consensus.cc:1261] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 1 FOLLOWER]: Refusing update from remote peer 3bea39b269ef4dc8a8819eef4ff0c5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:38.125247 1662 raft_consensus.cc:1261] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 1 FOLLOWER]: Refusing update from remote peer 3bea39b269ef4dc8a8819eef4ff0c5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:38.125551 1744 consensus_queue.cc:1035] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee844f0b498a43378010588db106096c" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:38.125708 1746 consensus_queue.cc:1035] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b5ce07e3ef4a6ab035b464b4042826" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:38.129096 1753 mvcc.cc:205] Tried to move back new op lower bound from 7316658495998410752 to 7316658495971205120. Current Snapshot: MvccSnapshot[applied={T|T < 7316658495998410752}]
I20260809 16:39:38.130573 1187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:38.130671 1187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:38.131469 1187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88cba194f0de4554b79c610ae5b750db: Sending DeleteTablet for 3 replicas of tablet 4e4ef824f14541b1b968c939e060f20e
I20260809 16:39:38.131785 1484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4ef824f14541b1b968c939e060f20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:34034
I20260809 16:39:38.131795 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4ef824f14541b1b968c939e060f20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50602
I20260809 16:39:38.131868 1333 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4ef824f14541b1b968c939e060f20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50316
I20260809 16:39:38.132103 1765 tablet_replica.cc:330] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826: stopping tablet replica
I20260809 16:39:38.132216 1765 raft_consensus.cc:2229] T 4e4ef824f14541b1b968c939e060f20e P 84b5ce07e3ef4a6ab035b464b4042826 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:38.132221 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1261
I20260809 16:39:38.134516 1763 tablet_replica.cc:330] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: stopping tablet replica
I20260809 16:39:38.134632 1763 raft_consensus.cc:2229] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:38.134758 1763 raft_consensus.cc:2258] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:38.134933 1762 tablet_replica.cc:330] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: stopping tablet replica
I20260809 16:39:38.135063 1762 raft_consensus.cc:2229] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:38.135162 1762 raft_consensus.cc:2258] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:38.135499 1763 ts_tablet_manager.cc:1905] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:38.135546 1762 ts_tablet_manager.cc:1905] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:38.138378 1762 ts_tablet_manager.cc:1918] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:38.138378 1763 ts_tablet_manager.cc:1918] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:38.138481 1762 log.cc:1196] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/4e4ef824f14541b1b968c939e060f20e
I20260809 16:39:38.138602 1763 log.cc:1196] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/4e4ef824f14541b1b968c939e060f20e
I20260809 16:39:38.138828 1762 ts_tablet_manager.cc:1939] T 4e4ef824f14541b1b968c939e060f20e P ee844f0b498a43378010588db106096c: Deleting consensus metadata
I20260809 16:39:38.138994 1763 ts_tablet_manager.cc:1939] T 4e4ef824f14541b1b968c939e060f20e P 3bea39b269ef4dc8a8819eef4ff0c5fd: Deleting consensus metadata
I20260809 16:39:38.139322 1173 catalog_manager.cc:4841] TS ee844f0b498a43378010588db106096c (127.28.62.131:34797): tablet 4e4ef824f14541b1b968c939e060f20e (table test-table [id=ade2802e412f475da98d6b2c7afd0ee0]) successfully deleted
I20260809 16:39:38.139565 1171 catalog_manager.cc:4841] TS 3bea39b269ef4dc8a8819eef4ff0c5fd (127.28.62.130:43259): tablet 4e4ef824f14541b1b968c939e060f20e (table test-table [id=ade2802e412f475da98d6b2c7afd0ee0]) successfully deleted
W20260809 16:39:38.142560 1171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:35239 (error 108)
I20260809 16:39:38.142928 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1414
I20260809 16:39:38.144156 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.143090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35239 (local address 127.0.0.1:50318)
0809 16:39:38.144074 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.144119 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":960}
W20260809 16:39:38.147586 1171 catalog_manager.cc:4584] TS 84b5ce07e3ef4a6ab035b464b4042826 (127.28.62.129:35239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e4ef824f14541b1b968c939e060f20e: Network error: Client connection negotiation failed: client connection to 127.28.62.129:35239: connect: Connection refused (error 111)
I20260809 16:39:38.151801 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1568
I20260809 16:39:38.159456 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1154
2026-08-09T16:39:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2008 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T16:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:38Z Disabled control of system clock
I20260809 16:39:38.191347 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42141
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 16:39:38.274196 1771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.274441 1771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.274462 1771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.275691 1771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:38.275735 1771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.275748 1771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.275760 1771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:38.275772 1771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:38.277109 1771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:42141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.277366 1771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.277668 1771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.282845 1779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.282832 1776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.283006 1771 server_base.cc:1034] running on GCE node
W20260809 16:39:38.282984 1777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.283414 1771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.283684 1771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.284825 1771 hybrid_clock.cc:648] HybridClock initialized: now 1786293578284809 us; error 32 us; skew 500 ppm
I20260809 16:39:38.286072 1771 webserver.cc:460] Webserver started at http://127.28.62.190:39075/ using document root <none> and password file <none>
I20260809 16:39:38.286314 1771 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.286360 1771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.286499 1771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.287597 1771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "c0ea8598ba4a4155ac7ce42f2713c776"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "b53fd47a30b540e022a9dd1ea8ef9b34"
server_key_iv: "f8123c7339e28f3ee5180d1c9f25291c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.287951 1771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "c0ea8598ba4a4155ac7ce42f2713c776"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "b53fd47a30b540e022a9dd1ea8ef9b34"
server_key_iv: "f8123c7339e28f3ee5180d1c9f25291c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.289260 1771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:38.289994 1786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.290263 1771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:38.290352 1771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "c0ea8598ba4a4155ac7ce42f2713c776"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "b53fd47a30b540e022a9dd1ea8ef9b34"
server_key_iv: "f8123c7339e28f3ee5180d1c9f25291c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.290421 1771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.293684 1771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.293956 1771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.294090 1771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.298233 1771 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:42141
I20260809 16:39:38.298269 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:42141 every 8 connection(s)
I20260809 16:39:38.298609 1771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:38.299352 1849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.301994 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: Bootstrap starting.
I20260809 16:39:38.302515 1849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.302753 1849 log.cc:824] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.303470 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: No bootstrap required, opened a new log
I20260809 16:39:38.304957 1849 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } }
I20260809 16:39:38.305089 1849 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.305112 1849 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ea8598ba4a4155ac7ce42f2713c776, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.305224 1849 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } }
I20260809 16:39:38.305274 1849 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:38.305294 1849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:38.305320 1849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.305738 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1771
I20260809 16:39:38.305851 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:38.305804 1849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } }
I20260809 16:39:38.305939 1849 leader_election.cc:302] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ea8598ba4a4155ac7ce42f2713c776; no voters:
I20260809 16:39:38.306080 28922 external_mini_cluster.cc:1428] Setting key 9f15fe501a9f6aca0883f73482c5b11e
I20260809 16:39:38.306156 1852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:38.306138 1849 leader_election.cc:288] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:38.306332 1852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [term 1 LEADER]: Becoming Leader. State: Replica: c0ea8598ba4a4155ac7ce42f2713c776, State: Running, Role: LEADER
I20260809 16:39:38.306412 1852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } }
I20260809 16:39:38.306478 1849 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:38.306746 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } } }
I20260809 16:39:38.306814 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:38.307287 1854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ea8598ba4a4155ac7ce42f2713c776. Latest consensus state: current_term: 1 leader_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ea8598ba4a4155ac7ce42f2713c776" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42141 } } }
I20260809 16:39:38.307426 1854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776 [sys.catalog]: This master's current role is: LEADER
W20260809 16:39:38.308377 1871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:39:38.308432 1871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:39:38.308494 1859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:38.308660 1859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:38.308944 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.306778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.28.62.190:42141)
0809 16:39:38.307106 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.307116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.307156 (+ 40us) server_negotiation.cc:407] Connection header received
0809 16:39:38.307285 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.307295 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.307342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.307486 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.308465 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.308468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.308468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.308472 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.308511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.308524 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.308626 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.308720 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.308772 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260809 16:39:38.310318 1859 catalog_manager.cc:1347] Generated new cluster ID: a81ec74e09e3466d85172f44add9b100
I20260809 16:39:38.310372 1859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:38.349383 1859 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:38.349601 1859 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:38.354004 1859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: Generated new TSK 0
I20260809 16:39:38.354193 1859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:39:38.358592 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--builtin_ntp_servers=127.28.62.148:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 16:39:38.441016 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.441222 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.441244 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.442298 1874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.442332 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.442358 1874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:38.443626 1874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.443892 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.444156 1874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.449118 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.449201 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.449159 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.449373 1874 server_base.cc:1034] running on GCE node
I20260809 16:39:38.449584 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.449819 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.450980 1874 hybrid_clock.cc:648] HybridClock initialized: now 1786293578450961 us; error 42 us; skew 500 ppm
I20260809 16:39:38.452350 1874 webserver.cc:460] Webserver started at http://127.28.62.129:44339/ using document root <none> and password file <none>
I20260809 16:39:38.452605 1874 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.452687 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.452849 1874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.453940 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "1f21ac308143455595142379d74dfd66"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "ede30535405cdeabbf4d150df5dfeed2"
server_key_iv: "0c1ecc94c1a38ffe6d171917e0985dae"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.454342 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "1f21ac308143455595142379d74dfd66"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "ede30535405cdeabbf4d150df5dfeed2"
server_key_iv: "0c1ecc94c1a38ffe6d171917e0985dae"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.455749 1874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:38.456502 1891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.456674 1874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:38.456779 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "1f21ac308143455595142379d74dfd66"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "ede30535405cdeabbf4d150df5dfeed2"
server_key_iv: "0c1ecc94c1a38ffe6d171917e0985dae"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.456853 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.460353 1874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.460623 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.460768 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.460979 1874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:38.461301 1874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:38.461356 1874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.461395 1874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:38.461424 1874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.467629 1874 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:35287
I20260809 16:39:38.467674 2024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:35287 every 8 connection(s)
I20260809 16:39:38.467999 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:38.471201 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.468705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:35417 (local address 127.28.62.190:42141)
0809 16:39:38.468769 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.468771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.469003 (+ 232us) server_negotiation.cc:407] Connection header received
0809 16:39:38.469599 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.469601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.469610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.469668 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.470887 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.470889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.470889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.470895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.470913 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.470917 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.470971 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.471111 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.471129 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:38.471400 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.468701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42141 (local address 127.28.62.129:35417)
0809 16:39:38.468867 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.468900 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.469466 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.469741 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.469744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.469752 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.470731 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.470731 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.470759 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.470787 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.470966 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.470971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.470975 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.471121 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.471183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260809 16:39:38.471901 2025 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42141
I20260809 16:39:38.471977 2025 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:38.472208 2025 heartbeater.cc:511] Master 127.28.62.190:42141 requested a full tablet report, sending...
I20260809 16:39:38.472729 1805 ts_manager.cc:194] Registered new tserver with Master: 1f21ac308143455595142379d74dfd66 (127.28.62.129:35287)
I20260809 16:39:38.474347 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 1874
I20260809 16:39:38.474458 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:38.474646 28922 external_mini_cluster.cc:1428] Setting key c7c92f1f6a76f48195673f27dff5c4f8
I20260809 16:39:38.475597 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--builtin_ntp_servers=127.28.62.148:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 16:39:38.557196 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.557399 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.557421 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.558584 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.558626 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.558651 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:38.560097 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.560357 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.560689 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.565968 2036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.565927 2034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.565940 2033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.566229 2028 server_base.cc:1034] running on GCE node
I20260809 16:39:38.566434 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.566646 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.567819 2028 hybrid_clock.cc:648] HybridClock initialized: now 1786293578567798 us; error 33 us; skew 500 ppm
I20260809 16:39:38.569207 2028 webserver.cc:460] Webserver started at http://127.28.62.130:42825/ using document root <none> and password file <none>
I20260809 16:39:38.569482 2028 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.569562 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.569734 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.570919 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "0cca3cc75e344cbc99a986da83fd4dc8"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "5f9c3030986aa39eec66ee32d9608df8"
server_key_iv: "7a26be05916cb533223eece0f84732bd"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.571372 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "0cca3cc75e344cbc99a986da83fd4dc8"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "5f9c3030986aa39eec66ee32d9608df8"
server_key_iv: "7a26be05916cb533223eece0f84732bd"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.572744 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:38.573508 2043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.573689 2028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:38.573798 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "0cca3cc75e344cbc99a986da83fd4dc8"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "5f9c3030986aa39eec66ee32d9608df8"
server_key_iv: "7a26be05916cb533223eece0f84732bd"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.573884 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.577661 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.577936 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.578091 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.578320 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:38.578646 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:38.578706 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.578749 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:38.578780 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.585099 2028 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:41403
I20260809 16:39:38.585172 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:41403 every 8 connection(s)
I20260809 16:39:38.585532 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:38.588433 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.586295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:57721 (local address 127.28.62.190:42141)
0809 16:39:38.586381 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.586384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.586574 (+ 190us) server_negotiation.cc:407] Connection header received
0809 16:39:38.587001 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.587002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.587049 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.587139 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.588213 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.588215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.588215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.588220 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.588236 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.588241 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.588300 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.588365 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.588378 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 16:39:38.588605 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.586247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42141 (local address 127.28.62.130:57721)
0809 16:39:38.586440 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.586480 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.586913 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.587190 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.587193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.587201 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.588072 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.588073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.588099 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.588124 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.588299 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.588302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.588304 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.588353 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.588398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260809 16:39:38.589066 2175 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42141
I20260809 16:39:38.589182 2175 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:38.589349 2175 heartbeater.cc:511] Master 127.28.62.190:42141 requested a full tablet report, sending...
I20260809 16:39:38.589717 1805 ts_manager.cc:194] Registered new tserver with Master: 0cca3cc75e344cbc99a986da83fd4dc8 (127.28.62.130:41403)
I20260809 16:39:38.591164 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2028
I20260809 16:39:38.591288 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:38.591501 28922 external_mini_cluster.cc:1428] Setting key 75b61a1ab24089b4c64cc418f34aa7d2
I20260809 16:39:38.592410 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--builtin_ntp_servers=127.28.62.148:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 16:39:38.673408 2178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.673666 2178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.673712 2178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.674881 2178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.674953 2178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.674998 2178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:38.676352 2178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.676638 2178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.676998 2178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.682072 2187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.682070 2184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.682273 2178 server_base.cc:1034] running on GCE node
W20260809 16:39:38.682147 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.682534 2178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.682730 2178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.683868 2178 hybrid_clock.cc:648] HybridClock initialized: now 1786293578683846 us; error 45 us; skew 500 ppm
I20260809 16:39:38.685189 2178 webserver.cc:460] Webserver started at http://127.28.62.131:35465/ using document root <none> and password file <none>
I20260809 16:39:38.685420 2178 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.685462 2178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.685549 2178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.686547 2178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "e65a7fba022e490a9612c1cd6e7b1b54"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "da78d9dd337f2b2c394e8881a56e2c7d"
server_key_iv: "1a61a7c10cca57d1c57cfa7a2c5119a6"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.686920 2178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "e65a7fba022e490a9612c1cd6e7b1b54"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "da78d9dd337f2b2c394e8881a56e2c7d"
server_key_iv: "1a61a7c10cca57d1c57cfa7a2c5119a6"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.688277 2178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:38.688930 2193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.689086 2178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:38.689162 2178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "e65a7fba022e490a9612c1cd6e7b1b54"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "da78d9dd337f2b2c394e8881a56e2c7d"
server_key_iv: "1a61a7c10cca57d1c57cfa7a2c5119a6"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.689224 2178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.692533 2178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.692746 2178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.692837 2178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.692992 2178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:38.693229 2178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:38.693260 2178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.693280 2178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:38.693292 2178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.698846 2178 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36637
I20260809 16:39:38.698925 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36637 every 8 connection(s)
I20260809 16:39:38.699185 2178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:38.702070 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.699923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:55087 (local address 127.28.62.190:42141)
0809 16:39:38.699997 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.699999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.700212 (+ 213us) server_negotiation.cc:407] Connection header received
0809 16:39:38.700634 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.700634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.700639 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.700705 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.701886 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.701887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.701887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.701890 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.701900 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.701904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.701942 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.702005 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.702017 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 16:39:38.702272 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.699848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42141 (local address 127.28.62.131:55087)
0809 16:39:38.700104 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.700132 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.700555 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.700758 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.700761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.700768 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.701759 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.701760 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.701787 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.701814 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.701948 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.701950 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.701952 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.702000 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.702044 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260809 16:39:38.702762 2329 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42141
I20260809 16:39:38.702862 2329 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:38.702976 2329 heartbeater.cc:511] Master 127.28.62.190:42141 requested a full tablet report, sending...
I20260809 16:39:38.703372 1805 ts_manager.cc:194] Registered new tserver with Master: e65a7fba022e490a9612c1cd6e7b1b54 (127.28.62.131:36637)
I20260809 16:39:38.708057 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2178
I20260809 16:39:38.708140 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:38.708279 28922 external_mini_cluster.cc:1428] Setting key f052f3f7195501061364a2ab8f440657
I20260809 16:39:38.709216 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:38.711602 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.710549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.28.62.190:42141)
0809 16:39:38.710620 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.710622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.710689 (+ 67us) server_negotiation.cc:407] Connection header received
0809 16:39:38.710748 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.710749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.710753 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.710799 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.711380 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.711381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.711381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.711385 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.711401 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.711404 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.711453 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.711486 (+ 33us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.711498 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:39:38.713274 1805 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:38.714723 1805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:38.718631 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.716850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36637 (local address 127.0.0.1:34110)
0809 16:39:38.717041 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.717083 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.717140 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.717630 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.717638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.717644 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.718127 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.718128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.718144 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.718169 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.718462 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.718469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.718472 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.718542 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.718555 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260809 16:39:38.718678 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.717238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.28.62.131:36637)
0809 16:39:38.717427 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.717430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.717438 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:38.717459 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.717460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.717467 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.717879 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.718349 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.718353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.718353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.718357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.718388 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.718402 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.718465 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.718574 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.718590 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260809 16:39:38.718731 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.716901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35287 (local address 127.0.0.1:34508)
0809 16:39:38.717221 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.717232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.717284 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.717799 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.717800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.717803 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.718288 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.718288 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.718295 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.718305 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.718528 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.718528 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.718529 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.718597 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.718610 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":97,"threads_started":1}
I20260809 16:39:38.719203 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.717025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.28.62.129:35287)
0809 16:39:38.717635 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.717639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.717649 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:38.717674 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.717676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.717684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.717964 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.718385 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.718393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.718393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.718398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.718433 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.718463 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.718530 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.718604 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.718619 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":75,"threads_started":1}
I20260809 16:39:38.719282 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.717188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.28.62.130:41403)
0809 16:39:38.717400 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.717405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.717516 (+ 111us) server_negotiation.cc:407] Connection header received
0809 16:39:38.717562 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.717565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.717576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.717648 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.718929 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.718935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.718935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.718939 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.718969 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.718985 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.719114 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.719171 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.719188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":108,"threads_started":1}
I20260809 16:39:38.719691 2250 tablet_service.cc:1467] Processing CreateTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 (DEFAULT_TABLE table=test-table [id=5089ad1fb5764c7d8fd1275e16d1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.719985 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d87cc21ed347f38250a9ad5e8297c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.719956 1953 tablet_service.cc:1467] Processing CreateTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 (DEFAULT_TABLE table=test-table [id=5089ad1fb5764c7d8fd1275e16d1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.720301 1953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d87cc21ed347f38250a9ad5e8297c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.722594 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.717084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:41403 (local address 127.0.0.1:49294)
0809 16:39:38.717442 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.717452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.717505 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.717970 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.717970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.717972 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.718837 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.718838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.718845 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.718859 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.719103 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.719104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.719105 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.719161 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.719174 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
I20260809 16:39:38.722580 2343 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Bootstrap starting.
I20260809 16:39:38.723002 2344 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: Bootstrap starting.
I20260809 16:39:38.723423 2343 tablet_bootstrap.cc:654] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.723354 2099 tablet_service.cc:1467] Processing CreateTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 (DEFAULT_TABLE table=test-table [id=5089ad1fb5764c7d8fd1275e16d1c6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:38.723587 2099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d87cc21ed347f38250a9ad5e8297c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.723670 2344 tablet_bootstrap.cc:654] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.723740 2343 log.cc:824] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.723978 2344 log.cc:824] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.724541 2343 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: No bootstrap required, opened a new log
I20260809 16:39:38.724586 2343 ts_tablet_manager.cc:1397] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:38.724722 2344 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: No bootstrap required, opened a new log
I20260809 16:39:38.724785 2344 ts_tablet_manager.cc:1397] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:38.726311 2343 raft_consensus.cc:348] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.726429 2343 raft_consensus.cc:374] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.726450 2343 raft_consensus.cc:729] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65a7fba022e490a9612c1cd6e7b1b54, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.726569 2343 consensus_queue.cc:260] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.726749 2349 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Bootstrap starting.
I20260809 16:39:38.726838 2343 ts_tablet_manager.cc:1428] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:38.726958 2329 heartbeater.cc:503] Master 127.28.62.190:42141 was elected leader, sending a full tablet report...
I20260809 16:39:38.727438 2344 raft_consensus.cc:348] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.727614 2344 raft_consensus.cc:374] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.727658 2344 raft_consensus.cc:729] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f21ac308143455595142379d74dfd66, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.727774 2344 consensus_queue.cc:260] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.728058 2344 ts_tablet_manager.cc:1428] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260809 16:39:38.728072 2025 heartbeater.cc:503] Master 127.28.62.190:42141 was elected leader, sending a full tablet report...
I20260809 16:39:38.728116 2349 tablet_bootstrap.cc:654] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.728449 2349 log.cc:824] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.729195 2349 tablet_bootstrap.cc:492] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: No bootstrap required, opened a new log
I20260809 16:39:38.729252 2349 ts_tablet_manager.cc:1397] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:38.731287 2349 raft_consensus.cc:348] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.731405 2349 raft_consensus.cc:374] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.731426 2349 raft_consensus.cc:729] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cca3cc75e344cbc99a986da83fd4dc8, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.731561 2349 consensus_queue.cc:260] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.731840 2349 ts_tablet_manager.cc:1428] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 16:39:38.731860 2175 heartbeater.cc:503] Master 127.28.62.190:42141 was elected leader, sending a full tablet report...
I20260809 16:39:38.746922 2350 raft_consensus.cc:482] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:38.747135 2350 raft_consensus.cc:504] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.747474 2350 leader_election.cc:288] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cca3cc75e344cbc99a986da83fd4dc8 (127.28.62.130:41403), 1f21ac308143455595142379d74dfd66 (127.28.62.129:35287)
I20260809 16:39:38.748795 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.747650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:41403 (local address 127.28.62.131:49603)
0809 16:39:38.747693 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.747706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.747744 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.747921 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.747922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.747925 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.748424 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.748424 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.748433 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.748448 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.748637 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.748638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.748639 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.748706 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.748723 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260809 16:39:38.748803 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.747732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:49603 (local address 127.28.62.130:41403)
0809 16:39:38.747781 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.747784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.747795 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:38.747824 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.747825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.747830 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.747891 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.748537 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.748539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.748539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.748544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.748561 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.748565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.748628 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.748725 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.748741 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:39:38.748906 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.747757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41731 (local address 127.28.62.129:35287)
0809 16:39:38.747815 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.747817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.748044 (+ 227us) server_negotiation.cc:407] Connection header received
0809 16:39:38.748114 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.748115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.748120 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.748165 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.748688 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.748689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.748690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.748694 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.748709 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.748713 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.748758 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.748819 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.748834 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:39:38.749120 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d87cc21ed347f38250a9ad5e8297c1" candidate_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" is_pre_election: true
I20260809 16:39:38.749235 2120 raft_consensus.cc:2454] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65a7fba022e490a9612c1cd6e7b1b54 in term 0.
I20260809 16:39:38.749379 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.747658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35287 (local address 127.28.62.131:41731)
0809 16:39:38.747979 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.747988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:38.748051 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:38.748183 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:38.748183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:38.748185 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:38.748606 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:39:38.748607 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:38.748614 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:39:38.748627 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:38.748766 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:38.748767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:38.748767 (+ 0us) client_negotiation.cc:769] Sending connection context
0809 16:39:38.748817 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:39:38.748832 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":137,"thread_start_us":134,"threads_started":1}
I20260809 16:39:38.749464 2196 leader_election.cc:302] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cca3cc75e344cbc99a986da83fd4dc8, e65a7fba022e490a9612c1cd6e7b1b54; no voters:
I20260809 16:39:38.749575 2350 raft_consensus.cc:2781] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:38.749660 2350 raft_consensus.cc:482] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:38.749683 2350 raft_consensus.cc:3037] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.749760 1976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d87cc21ed347f38250a9ad5e8297c1" candidate_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f21ac308143455595142379d74dfd66" is_pre_election: true
I20260809 16:39:38.749913 1976 raft_consensus.cc:2454] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65a7fba022e490a9612c1cd6e7b1b54 in term 0.
I20260809 16:39:38.750594 2350 raft_consensus.cc:504] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.750733 2350 leader_election.cc:288] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [CANDIDATE]: Term 1 election: Requested vote from peers 0cca3cc75e344cbc99a986da83fd4dc8 (127.28.62.130:41403), 1f21ac308143455595142379d74dfd66 (127.28.62.129:35287)
I20260809 16:39:38.750859 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d87cc21ed347f38250a9ad5e8297c1" candidate_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cca3cc75e344cbc99a986da83fd4dc8"
I20260809 16:39:38.750860 1976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d87cc21ed347f38250a9ad5e8297c1" candidate_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f21ac308143455595142379d74dfd66"
I20260809 16:39:38.750941 1976 raft_consensus.cc:3037] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.750941 2120 raft_consensus.cc:3037] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.751878 2120 raft_consensus.cc:2454] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65a7fba022e490a9612c1cd6e7b1b54 in term 1.
I20260809 16:39:38.751878 1976 raft_consensus.cc:2454] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65a7fba022e490a9612c1cd6e7b1b54 in term 1.
I20260809 16:39:38.752091 2196 leader_election.cc:302] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cca3cc75e344cbc99a986da83fd4dc8, e65a7fba022e490a9612c1cd6e7b1b54; no voters:
I20260809 16:39:38.752238 2350 raft_consensus.cc:2781] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:38.752427 2350 raft_consensus.cc:686] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 1 LEADER]: Becoming Leader. State: Replica: e65a7fba022e490a9612c1cd6e7b1b54, State: Running, Role: LEADER
I20260809 16:39:38.752521 2350 consensus_queue.cc:237] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } }
I20260809 16:39:38.753381 1805 catalog_manager.cc:5495] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to e65a7fba022e490a9612c1cd6e7b1b54 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e65a7fba022e490a9612c1cd6e7b1b54" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:38.756963 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.756098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.28.62.131:36637)
0809 16:39:38.756147 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.756149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.756158 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:38.756190 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.756191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.756203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.756250 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.756751 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.756752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.756752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.756757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.756769 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.756773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.756831 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.756886 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.756900 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:38.758541 1976 raft_consensus.cc:1261] T a8d87cc21ed347f38250a9ad5e8297c1 P 1f21ac308143455595142379d74dfd66 [term 1 FOLLOWER]: Refusing update from remote peer e65a7fba022e490a9612c1cd6e7b1b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:38.758540 2120 raft_consensus.cc:1261] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 1 FOLLOWER]: Refusing update from remote peer e65a7fba022e490a9612c1cd6e7b1b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:38.758855 2356 consensus_queue.cc:1035] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f21ac308143455595142379d74dfd66" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:38.758957 2350 consensus_queue.cc:1035] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cca3cc75e344cbc99a986da83fd4dc8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:38.761205 2359 mvcc.cc:205] Tried to move back new op lower bound from 7316658498592788480 to 7316658498571251712. Current Snapshot: MvccSnapshot[applied={T|T < 7316658498592788480}]
I20260809 16:39:38.761492 2361 mvcc.cc:205] Tried to move back new op lower bound from 7316658498592788480 to 7316658498571251712. Current Snapshot: MvccSnapshot[applied={T|T < 7316658498592788480}]
I20260809 16:39:38.761691 2360 mvcc.cc:205] Tried to move back new op lower bound from 7316658498592788480 to 7316658498571251712. Current Snapshot: MvccSnapshot[applied={T|T < 7316658498592788480}]
I20260809 16:39:38.764616 1805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:38.764727 1805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:38.765442 1805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ea8598ba4a4155ac7ce42f2713c776: Sending DeleteTablet for 3 replicas of tablet a8d87cc21ed347f38250a9ad5e8297c1
I20260809 16:39:38.765758 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='slave'} at 127.0.0.1:34110
I20260809 16:39:38.765811 2099 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='slave'} at 127.0.0.1:49294
I20260809 16:39:38.765802 1953 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d87cc21ed347f38250a9ad5e8297c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:38 UTC) from {username='slave'} at 127.0.0.1:34508
I20260809 16:39:38.766222 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1874
I20260809 16:39:38.766367 2373 tablet_replica.cc:330] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: stopping tablet replica
I20260809 16:39:38.766474 2373 raft_consensus.cc:2229] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:38.766573 2373 raft_consensus.cc:2258] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:38.768065 2373 ts_tablet_manager.cc:1905] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:38.769651 2372 tablet_replica.cc:330] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: stopping tablet replica
I20260809 16:39:38.769697 2373 ts_tablet_manager.cc:1918] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:38.769762 2372 raft_consensus.cc:2229] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:38.769778 2373 log.cc:1196] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/a8d87cc21ed347f38250a9ad5e8297c1
I20260809 16:39:38.769893 2372 raft_consensus.cc:2258] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:38.770048 2373 ts_tablet_manager.cc:1939] T a8d87cc21ed347f38250a9ad5e8297c1 P 0cca3cc75e344cbc99a986da83fd4dc8: Deleting consensus metadata
I20260809 16:39:38.770349 2372 ts_tablet_manager.cc:1905] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:38.770448 1790 catalog_manager.cc:4841] TS 0cca3cc75e344cbc99a986da83fd4dc8 (127.28.62.130:41403): tablet a8d87cc21ed347f38250a9ad5e8297c1 (table test-table [id=5089ad1fb5764c7d8fd1275e16d1c6e9]) successfully deleted
I20260809 16:39:38.772360 2372 ts_tablet_manager.cc:1918] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:38.772442 2372 log.cc:1196] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/a8d87cc21ed347f38250a9ad5e8297c1
I20260809 16:39:38.772712 2372 ts_tablet_manager.cc:1939] T a8d87cc21ed347f38250a9ad5e8297c1 P e65a7fba022e490a9612c1cd6e7b1b54: Deleting consensus metadata
I20260809 16:39:38.773160 1790 catalog_manager.cc:4841] TS e65a7fba022e490a9612c1cd6e7b1b54 (127.28.62.131:36637): tablet a8d87cc21ed347f38250a9ad5e8297c1 (table test-table [id=5089ad1fb5764c7d8fd1275e16d1c6e9]) successfully deleted
W20260809 16:39:38.775238 1791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:35287 (error 108)
I20260809 16:39:38.775516 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2028
I20260809 16:39:38.776199 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.775685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:35287 (local address 127.0.0.1:34518)
0809 16:39:38.776100 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:38.776148 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260809 16:39:38.782932 1791 catalog_manager.cc:4584] TS 1f21ac308143455595142379d74dfd66 (127.28.62.129:35287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8d87cc21ed347f38250a9ad5e8297c1: Network error: Client connection negotiation failed: client connection to 127.28.62.129:35287: connect: Connection refused (error 111)
I20260809 16:39:38.789201 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2178
I20260809 16:39:38.797446 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 1771
2026-08-09T16:39:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (636 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 16:39:38.819957 28922 mini_oidc.cc:152] Starting JWKS server
I20260809 16:39:38.821195 28922 webserver.cc:460] Webserver started at https://127.0.0.1:42673/ using document root <none> and password file <none>
I20260809 16:39:38.821322 28922 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 16:39:38.821511 28922 webserver.cc:460] Webserver started at http://127.0.0.1:37071/ using document root <none> and password file <none>
2026-08-09T16:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:38Z Disabled control of system clock
I20260809 16:39:38.830394 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44131
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44131
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42673/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/testchainca.cert with env {}
W20260809 16:39:38.912111 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:38.912333 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:38.912356 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:38.913555 2383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:38.913601 2383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:38.913615 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:38.913627 2383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:38.913640 2383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 16:39:38.913652 2383 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 16:39:38.913712 2383 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42673/jwks.json
I20260809 16:39:38.915385 2383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42673/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:38.915638 2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:38.915900 2383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:38.920905 2389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:38.920881 2391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.921099 2383 server_base.cc:1034] running on GCE node
W20260809 16:39:38.920934 2388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:38.921434 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:38.921706 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:38.922880 2383 hybrid_clock.cc:648] HybridClock initialized: now 1786293578922863 us; error 30 us; skew 500 ppm
I20260809 16:39:38.924523 2383 webserver.cc:460] Webserver started at http://127.28.62.190:36519/ using document root <none> and password file <none>
I20260809 16:39:38.924849 2383 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:38.924904 2383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:38.925056 2383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:38.926313 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "ecfec966620a4aeb86a3e5c0461f7286"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "4e09f59f22805df2dc15bdfdf05a4b35"
server_key_iv: "492a0fed0bd1b41508b683739e9e492c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.926738 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "ecfec966620a4aeb86a3e5c0461f7286"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "4e09f59f22805df2dc15bdfdf05a4b35"
server_key_iv: "492a0fed0bd1b41508b683739e9e492c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.928458 2383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260809 16:39:38.929497 2398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:38.929737 2383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:38.929845 2383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "ecfec966620a4aeb86a3e5c0461f7286"
format_stamp: "Formatted at 2026-08-09 16:39:38 on dist-test-slave-wl2h"
server_key: "4e09f59f22805df2dc15bdfdf05a4b35"
server_key_iv: "492a0fed0bd1b41508b683739e9e492c"
server_key_version: "encryptionkey@0"
I20260809 16:39:38.929924 2383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:38.942898 2383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:38.943284 2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:38.943403 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:38.947906 2383 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44131
I20260809 16:39:38.947988 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44131 every 8 connection(s)
I20260809 16:39:38.948403 2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:38.949542 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:38.951934 2461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286: Bootstrap starting.
I20260809 16:39:38.952582 2461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:38.952903 2461 log.cc:824] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:38.953675 2461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286: No bootstrap required, opened a new log
I20260809 16:39:38.955140 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2383
I20260809 16:39:38.955230 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:38.955418 28922 external_mini_cluster.cc:1428] Setting key 6423dfb508aa77d8f63f97d7da70611f
I20260809 16:39:38.955870 2461 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } }
I20260809 16:39:38.956184 2461 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:38.956260 2461 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfec966620a4aeb86a3e5c0461f7286, State: Initialized, Role: FOLLOWER
I20260809 16:39:38.956413 2461 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } }
I20260809 16:39:38.956526 2461 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:38.956579 2461 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:38.956638 2461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:38.957250 2461 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } }
I20260809 16:39:38.957398 2461 leader_election.cc:302] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecfec966620a4aeb86a3e5c0461f7286; no voters:
I20260809 16:39:38.957626 2461 leader_election.cc:288] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:38.957669 2466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:38.957827 2466 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [term 1 LEADER]: Becoming Leader. State: Replica: ecfec966620a4aeb86a3e5c0461f7286, State: Running, Role: LEADER
I20260809 16:39:38.957940 2461 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:38.957940 2466 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } }
I20260809 16:39:38.958348 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecfec966620a4aeb86a3e5c0461f7286. Latest consensus state: current_term: 1 leader_uuid: "ecfec966620a4aeb86a3e5c0461f7286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } } }
I20260809 16:39:38.958424 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:38.958359 2468 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecfec966620a4aeb86a3e5c0461f7286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecfec966620a4aeb86a3e5c0461f7286" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44131 } } }
I20260809 16:39:38.958652 2468 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:38.959456 2471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:38.960048 2471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:38.962167 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:38.956030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.28.62.190:44131)
0809 16:39:38.956280 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:38.956287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:38.956338 (+ 51us) server_negotiation.cc:407] Connection header received
0809 16:39:38.956463 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:38.956468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:38.956522 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:38.956673 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:38.956912 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.958508 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.959667 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:38.960319 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:38.960444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:38.961332 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:38.961341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:38.961341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:38.961602 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:38.961652 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:38.961664 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:38.961749 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:38.961872 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 16:39:38.961956 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260809 16:39:38.963370 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:39:38.971500 2471 catalog_manager.cc:1347] Generated new cluster ID: b80e93f123a5473fb721a3b6c686e26c
I20260809 16:39:38.971614 2471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:38.985478 2471 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:38.986073 2471 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:38.995810 2471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecfec966620a4aeb86a3e5c0461f7286: Generated new TSK 0
I20260809 16:39:38.996053 2471 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 16:39:39.026619 2491 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 16:39:39.026784 2491 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 16:39:39.026875 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.003883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.28.62.190:44131)
0809 16:39:39.003938 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.003940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.004175 (+ 235us) server_negotiation.cc:407] Connection header received
0809 16:39:39.004215 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.004216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.004228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.004282 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0809 16:39:39.004465 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.004956 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.006129 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.006315 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.006375 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.006625 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 16:39:39.026564 (+ 19939us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 16:39:39.026780 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 16:39:39.026937 2465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 16:39:39.027333 2490 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:39.003749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44131 (local address 127.0.0.1:58616)
0809 16:39:39.004021 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:39.004033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:39.004109 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:39.004313 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:39.004314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:39.004316 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0809 16:39:39.004390 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.004393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.005057 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.005059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.005997 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.006001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.006412 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.006413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.006487 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.006509 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 16:39:39.026741 (+ 20232us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 16:39:39.026877 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.62.190:44131: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.62.190:44131: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42673/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 16:39:39.027792 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2383
2026-08-09T16:39:39Z chronyd exiting
I20260809 16:39:39.067821 28922 test_util.cc:175] -----------------------------------------------
I20260809 16:39:39.067931 28922 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786293555794774-28922-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (249 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 16:39:39.068750 28922 mini_oidc.cc:152] Starting JWKS server
I20260809 16:39:39.069468 28922 webserver.cc:460] Webserver started at https://127.0.0.1:38373/ using document root <none> and password file <none>
I20260809 16:39:39.069536 28922 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 16:39:39.069705 28922 webserver.cc:460] Webserver started at http://127.0.0.1:37457/ using document root <none> and password file <none>
2026-08-09T16:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:39Z Disabled control of system clock
I20260809 16:39:39.078750 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36893
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:36893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38373/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/testchainca.cert with env {}
W20260809 16:39:39.161065 2498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:39.161360 2498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:39.161401 2498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:39.162563 2498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:39.162632 2498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:39.162654 2498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:39.162675 2498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:39.162705 2498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 16:39:39.162734 2498 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 16:39:39.162756 2498 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38373/jwks.json
I20260809 16:39:39.164309 2498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:36893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38373/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:39.164597 2498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:39.164819 2498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:39.169888 2503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:39.169894 2506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:39.170059 2498 server_base.cc:1034] running on GCE node
W20260809 16:39:39.169902 2504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:39.170487 2498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:39.170801 2498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:39.171994 2498 hybrid_clock.cc:648] HybridClock initialized: now 1786293579171969 us; error 38 us; skew 500 ppm
I20260809 16:39:39.173439 2498 webserver.cc:460] Webserver started at http://127.28.62.190:35373/ using document root <none> and password file <none>
I20260809 16:39:39.173694 2498 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:39.173771 2498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:39.173923 2498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:39.175231 2498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "fa1643f64d204c059c581fdc073da925"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "74cfca1cad7985310dfc4257476372e3"
server_key_iv: "18a094eb224ffbd1f04eda8e45522cbf"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.175719 2498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "fa1643f64d204c059c581fdc073da925"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "74cfca1cad7985310dfc4257476372e3"
server_key_iv: "18a094eb224ffbd1f04eda8e45522cbf"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.177294 2498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:39:39.178081 2513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:39.178263 2498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:39.178365 2498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "fa1643f64d204c059c581fdc073da925"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "74cfca1cad7985310dfc4257476372e3"
server_key_iv: "18a094eb224ffbd1f04eda8e45522cbf"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.178447 2498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:39.195577 2498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:39.195935 2498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:39.196097 2498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:39.200603 2498 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:36893
I20260809 16:39:39.200639 2577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:36893 every 8 connection(s)
I20260809 16:39:39.200997 2498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:39.201977 2578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:39.203603 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2498
I20260809 16:39:39.203709 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:39.203948 28922 external_mini_cluster.cc:1428] Setting key 5ee5e0368753af1b27d6687d6d4958c9
I20260809 16:39:39.205783 2578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925: Bootstrap starting.
I20260809 16:39:39.206570 2578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:39.206898 2578 log.cc:824] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:39.207760 2578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925: No bootstrap required, opened a new log
I20260809 16:39:39.209900 2578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } }
I20260809 16:39:39.210103 2578 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:39.210163 2578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1643f64d204c059c581fdc073da925, State: Initialized, Role: FOLLOWER
I20260809 16:39:39.210242 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.204520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.28.62.190:36893)
0809 16:39:39.204764 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.204773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.204808 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:39:39.205465 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.205474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.205560 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.205731 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:39.205879 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.207287 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.208193 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.208606 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.208756 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.209541 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:39.209546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:39.209546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:39.209713 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:39.209755 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:39.209765 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:39.209847 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:39.209923 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 16:39:39.210014 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260809 16:39:39.210317 2578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } }
I20260809 16:39:39.210417 2578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:39.210457 2578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:39.210507 2578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:39.211295 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:39:39.211511 2578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } }
I20260809 16:39:39.211614 2578 leader_election.cc:302] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa1643f64d204c059c581fdc073da925; no voters:
I20260809 16:39:39.211792 2578 leader_election.cc:288] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:39.211848 2584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:39.212000 2584 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [term 1 LEADER]: Becoming Leader. State: Replica: fa1643f64d204c059c581fdc073da925, State: Running, Role: LEADER
I20260809 16:39:39.212085 2578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:39.212121 2584 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } }
I20260809 16:39:39.212466 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa1643f64d204c059c581fdc073da925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } } }
I20260809 16:39:39.212456 2584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa1643f64d204c059c581fdc073da925. Latest consensus state: current_term: 1 leader_uuid: "fa1643f64d204c059c581fdc073da925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1643f64d204c059c581fdc073da925" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36893 } } }
I20260809 16:39:39.212551 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.212559 2584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.212790 2591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:39.213366 2591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:39.215094 2591 catalog_manager.cc:1347] Generated new cluster ID: d0905b3052794d31a1fa2366539d6c27
I20260809 16:39:39.215153 2591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:39.243052 2591 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:39.243844 2591 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:39.250226 2591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa1643f64d204c059c581fdc073da925: Generated new TSK 0
I20260809 16:39:39.250461 2591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 16:39:39.275187 2615 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 16:39:39.275295 2615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 16:39:39.275580 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.260331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.28.62.190:36893)
0809 16:39:39.260380 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.260383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.260481 (+ 98us) server_negotiation.cc:407] Connection header received
0809 16:39:39.260554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.260555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.260570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.260648 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0809 16:39:39.260780 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.261585 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.262466 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.262790 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.262864 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.263058 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 16:39:39.275337 (+ 12279us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 16:39:39.275505 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 16:39:39.275648 2582 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 16:39:39.275684 2613 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:39.260227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:36893 (local address 127.0.0.1:60996)
0809 16:39:39.260395 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:39.260407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:39.260476 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:39.260653 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:39.260654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:39.260655 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0809 16:39:39.260713 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.260716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.261710 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.261711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.262371 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.262374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.262875 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.262876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.262922 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.262927 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 16:39:39.275486 (+ 12559us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 16:39:39.275581 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.62.190:36893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260809 16:39:39.276094 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2498
2026-08-09T16:39:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 16:39:39.317591 28922 mini_oidc.cc:152] Starting JWKS server
I20260809 16:39:39.318326 28922 webserver.cc:460] Webserver started at https://127.0.0.1:46335/ using document root <none> and password file <none>
I20260809 16:39:39.318403 28922 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 16:39:39.318595 28922 webserver.cc:460] Webserver started at http://127.0.0.1:42223/ using document root <none> and password file <none>
2026-08-09T16:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:39Z Disabled control of system clock
I20260809 16:39:39.328025 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42865
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:42865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46335/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 16:39:39.416890 2622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:39.417148 2622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:39.417186 2622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:39.418427 2622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:39.418494 2622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:39.418519 2622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:39.418552 2622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:39.418574 2622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 16:39:39.418596 2622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 16:39:39.418622 2622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46335/jwks.json
I20260809 16:39:39.420125 2622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:42865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46335/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:39.420418 2622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:39.420729 2622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:39.425871 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:39.425861 2629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:39.425979 2622 server_base.cc:1034] running on GCE node
W20260809 16:39:39.425864 2631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:39.426440 2622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:39.426808 2622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:39.427961 2622 hybrid_clock.cc:648] HybridClock initialized: now 1786293579427924 us; error 58 us; skew 500 ppm
I20260809 16:39:39.429281 2622 webserver.cc:460] Webserver started at http://127.28.62.190:44173/ using document root <none> and password file <none>
I20260809 16:39:39.429548 2622 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:39.429622 2622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:39.429778 2622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:39.430882 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "7ac1b0c20b1457b33e0a24c4dafbcf7b"
server_key_iv: "e172b971a2ecf8b0610a350f14cb1486"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.431321 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "7ac1b0c20b1457b33e0a24c4dafbcf7b"
server_key_iv: "e172b971a2ecf8b0610a350f14cb1486"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.432669 2622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:39.433439 2638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:39.433651 2622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:39.433780 2622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "7ac1b0c20b1457b33e0a24c4dafbcf7b"
server_key_iv: "e172b971a2ecf8b0610a350f14cb1486"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.433857 2622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:39.443849 2622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:39.444159 2622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:39.444326 2622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:39.448841 2622 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:42865
I20260809 16:39:39.448882 2696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:42865 every 8 connection(s)
I20260809 16:39:39.449241 2622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:39.449955 2698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:39.452759 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6: Bootstrap starting.
I20260809 16:39:39.453375 2698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:39.453677 2698 log.cc:824] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:39.454384 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6: No bootstrap required, opened a new log
I20260809 16:39:39.456112 2698 raft_consensus.cc:348] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } }
I20260809 16:39:39.456256 2698 raft_consensus.cc:374] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:39.456279 2698 raft_consensus.cc:729] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72a0e3a30444938aeb8ac4b3cfb12c6, State: Initialized, Role: FOLLOWER
I20260809 16:39:39.456465 2698 consensus_queue.cc:260] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } }
I20260809 16:39:39.456529 2698 raft_consensus.cc:388] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:39.456585 2698 raft_consensus.cc:482] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:39.456637 2698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:39.457134 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2622
I20260809 16:39:39.457248 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:39.457207 2698 raft_consensus.cc:504] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } }
I20260809 16:39:39.457334 2698 leader_election.cc:302] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d72a0e3a30444938aeb8ac4b3cfb12c6; no voters:
I20260809 16:39:39.457472 28922 external_mini_cluster.cc:1428] Setting key 50eb9ae8213e7d9914200eeef0d1e551
I20260809 16:39:39.457520 2698 leader_election.cc:288] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:39.457573 2701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:39.457729 2701 raft_consensus.cc:686] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [term 1 LEADER]: Becoming Leader. State: Replica: d72a0e3a30444938aeb8ac4b3cfb12c6, State: Running, Role: LEADER
I20260809 16:39:39.457787 2698 sys_catalog.cc:564] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:39.457836 2701 consensus_queue.cc:237] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } }
I20260809 16:39:39.458214 2703 sys_catalog.cc:455] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } } }
I20260809 16:39:39.458251 2701 sys_catalog.cc:455] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d72a0e3a30444938aeb8ac4b3cfb12c6. Latest consensus state: current_term: 1 leader_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72a0e3a30444938aeb8ac4b3cfb12c6" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42865 } } }
I20260809 16:39:39.458375 2703 sys_catalog.cc:458] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.458392 2701 sys_catalog.cc:458] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.458803 2714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:39.459371 2714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:39.460918 2714 catalog_manager.cc:1347] Generated new cluster ID: bc1e512b92d946ed8a541f008f96a382
I20260809 16:39:39.460973 2714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:39.464234 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.458177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.28.62.190:42865)
0809 16:39:39.458372 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.458381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.458641 (+ 260us) server_negotiation.cc:407] Connection header received
0809 16:39:39.458791 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.458800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.458881 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.459042 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:39.459361 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.460967 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.462090 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.462683 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.462803 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.463537 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:39.463540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:39.463540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:39.463748 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:39.463787 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:39.463801 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:39.463864 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:39.463962 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 16:39:39.464033 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260809 16:39:39.465178 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:39:39.466951 2714 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:39.467440 2714 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:39.476125 2714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d72a0e3a30444938aeb8ac4b3cfb12c6: Generated new TSK 0
I20260809 16:39:39.476312 2714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:39:39.487793 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.484301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.28.62.190:42865)
0809 16:39:39.484370 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.484372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.484453 (+ 81us) server_negotiation.cc:407] Connection header received
0809 16:39:39.484554 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.484555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.484568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.484657 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0809 16:39:39.484819 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.485338 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.486116 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.486352 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.486431 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.486620 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 16:39:39.487564 (+ 944us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46335/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 16:39:39.487701 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46335/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 16:39:39.487856 2708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46335/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 16:39:39.487902 2733 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:39.484220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42865 (local address 127.0.0.1:56086)
0809 16:39:39.484378 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:39.484389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:39.484471 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:39.484655 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:39.484656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:39.484658 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0809 16:39:39.484740 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.484742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.485426 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.485427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.486026 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.486027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.486436 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.486437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.486506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.486511 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 16:39:39.487672 (+ 1161us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46335/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 16:39:39.487791 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.62.190:42865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46335/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260809 16:39:39.488452 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2622
E20260809 16:39:39.502034 2734 webserver.cc:533] Webserver: sslize failed:
E20260809 16:39:39.502120 2734 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T16:39:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T16:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:39Z Disabled control of system clock
I20260809 16:39:39.539541 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:43343
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:43343
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/testchainca.cert with env {}
W20260809 16:39:39.623587 2738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:39.623802 2738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:39.623826 2738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:39.625026 2738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:39.625074 2738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:39.625087 2738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:39.625099 2738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:39.625113 2738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 16:39:39.625125 2738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 16:39:39.625195 2738 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 16:39:39.626850 2738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:43343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:39.627324 2738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:39.627589 2738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:39.633047 2746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:39.633059 2748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:39.633059 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:39.633518 2738 server_base.cc:1034] running on GCE node
I20260809 16:39:39.633790 2738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:39.634090 2738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:39.635260 2738 hybrid_clock.cc:648] HybridClock initialized: now 1786293579635241 us; error 32 us; skew 500 ppm
I20260809 16:39:39.636755 2738 webserver.cc:460] Webserver started at http://127.28.62.190:45803/ using document root <none> and password file <none>
I20260809 16:39:39.636960 2738 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:39.637003 2738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:39.637099 2738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:39.638147 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "90a6419d1f5c420daad7c15d8191ac7c"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "35bb8edaaa83c2ba9b9c8e5fca3e0102"
server_key_iv: "35c8f4179cc00d2d6b613402265085cb"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.638535 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "90a6419d1f5c420daad7c15d8191ac7c"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "35bb8edaaa83c2ba9b9c8e5fca3e0102"
server_key_iv: "35c8f4179cc00d2d6b613402265085cb"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.640004 2738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:39.640919 2755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:39.641304 2738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:39.641450 2738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "90a6419d1f5c420daad7c15d8191ac7c"
format_stamp: "Formatted at 2026-08-09 16:39:39 on dist-test-slave-wl2h"
server_key: "35bb8edaaa83c2ba9b9c8e5fca3e0102"
server_key_iv: "35c8f4179cc00d2d6b613402265085cb"
server_key_version: "encryptionkey@0"
I20260809 16:39:39.641543 2738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:39.655872 2738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:39.656246 2738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:39.656418 2738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:39.660693 2738 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:43343
I20260809 16:39:39.660754 2815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:43343 every 8 connection(s)
I20260809 16:39:39.661134 2738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:39.662264 2816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:39.664407 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2738
I20260809 16:39:39.664539 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:39.664719 28922 external_mini_cluster.cc:1428] Setting key 1f91a4f080a9e890b1b6a475e0142b28
I20260809 16:39:39.664811 2816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c: Bootstrap starting.
I20260809 16:39:39.665577 2816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:39.665937 2816 log.cc:824] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:39.666744 2816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c: No bootstrap required, opened a new log
I20260809 16:39:39.669066 2816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } }
I20260809 16:39:39.669242 2816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:39.669279 2816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a6419d1f5c420daad7c15d8191ac7c, State: Initialized, Role: FOLLOWER
I20260809 16:39:39.669397 2816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } }
I20260809 16:39:39.669485 2816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:39.669528 2816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:39.669575 2816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:39.670176 2816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } }
I20260809 16:39:39.670295 2816 leader_election.cc:302] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a6419d1f5c420daad7c15d8191ac7c; no voters:
I20260809 16:39:39.670499 2816 leader_election.cc:288] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:39.670523 2821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:39.670641 2821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [term 1 LEADER]: Becoming Leader. State: Replica: 90a6419d1f5c420daad7c15d8191ac7c, State: Running, Role: LEADER
I20260809 16:39:39.670737 2821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } }
I20260809 16:39:39.670789 2816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:39.671211 2821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a6419d1f5c420daad7c15d8191ac7c. Latest consensus state: current_term: 1 leader_uuid: "90a6419d1f5c420daad7c15d8191ac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } } }
I20260809 16:39:39.671324 2821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.671224 2822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90a6419d1f5c420daad7c15d8191ac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a6419d1f5c420daad7c15d8191ac7c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43343 } } }
I20260809 16:39:39.671487 2822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:39.671501 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.665287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.28.62.190:43343)
0809 16:39:39.665542 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.665551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.665587 (+ 36us) server_negotiation.cc:407] Connection header received
0809 16:39:39.666233 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.666240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.666305 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.666466 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:39.666641 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.668015 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.668861 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.669483 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.669613 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.670389 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:39.670393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:39.670393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:39:39.670609 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:39.670653 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:39:39.670669 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:39.670918 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:39.670949 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 16:39:39.671084 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260809 16:39:39.671558 2828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:39.672079 2828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:39.672744 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:39:39.674183 2828 catalog_manager.cc:1347] Generated new cluster ID: 7e7380dfef7b45a98da07c9b9503c0f2
I20260809 16:39:39.674240 2828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:39.683917 2828 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:39.684624 2828 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:39.690092 2828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90a6419d1f5c420daad7c15d8191ac7c: Generated new TSK 0
I20260809 16:39:39.690274 2828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:39:40.151525 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:39.698957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.28.62.190:43343)
0809 16:39:39.699006 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:39.699026 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:39.699165 (+ 139us) server_negotiation.cc:407] Connection header received
0809 16:39:39.699222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:39.699223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:39.699235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:39.699304 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0809 16:39:39.699452 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.700245 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.701061 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.701391 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.701464 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.701639 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 16:39:40.151236 (+449597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 16:39:40.151457 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35006: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260809 16:39:40.151614 2847 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:39.698884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:43343 (local address 127.0.0.1:35006)
0809 16:39:39.699084 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:39.699095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:39.699156 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:39.699319 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:39.699320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:39.699322 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0809 16:39:39.699380 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.699382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.700374 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.700374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.700979 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:39.700981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:39.701483 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:39.701484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:39.701529 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:39.701534 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 16:39:40.151395 (+449861us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 16:39:40.151522 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.62.190:43343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":116,"threads_started":1}
W20260809 16:39:40.151639 2819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35006: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260809 16:39:40.151888 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2738
2026-08-09T16:39:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (643 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (98 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:41 dist-test-slave-wl2h krb5kdc[2860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:41 dist-test-slave-wl2h krb5kdc[2860](info): set up 2 sockets
Aug 09 16:39:41 dist-test-slave-wl2h krb5kdc[2860](info): commencing operation
krb5kdc: starting...
W20260809 16:39:42.496181 28922 mini_kdc.cc:121] Time spent starting KDC: real 2.130s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:42 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:42.635116 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35575
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:42.721021 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:42.721312 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:42.721355 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:42.722571 2876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:42.722653 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:42.722678 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:42.722699 2876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:42.722730 2876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:42.724313 2876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:42.724637 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:42.724890 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:42.730207 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:42.730459 2876 server_base.cc:1034] running on GCE node
W20260809 16:39:42.730239 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:42.730219 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:42.730842 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:42.731172 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:42.732333 2876 hybrid_clock.cc:648] HybridClock initialized: now 1786293582732320 us; error 30 us; skew 500 ppm
Aug 09 16:39:42 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:42.739189 2876 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:42.739812 2876 webserver.cc:460] Webserver started at http://127.28.62.190:42967/ using document root <none> and password file <none>
I20260809 16:39:42.740021 2876 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:42.740063 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:42.740168 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:42.741330 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "6770ec236a7a47d18d253b71e4f0696f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "13adfedb7530a0ee2b72cb0a345fc99a"
server_key_iv: "4f562f173ab2f001f3429ef5187ba99c"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.741657 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "6770ec236a7a47d18d253b71e4f0696f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "13adfedb7530a0ee2b72cb0a345fc99a"
server_key_iv: "4f562f173ab2f001f3429ef5187ba99c"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.743177 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:42.744127 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:42.744700 2876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:42.744853 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "6770ec236a7a47d18d253b71e4f0696f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "13adfedb7530a0ee2b72cb0a345fc99a"
server_key_iv: "4f562f173ab2f001f3429ef5187ba99c"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.744947 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:42.764544 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:42.767400 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:42.767606 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:42.771970 2876 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35575
I20260809 16:39:42.772027 2954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35575 every 8 connection(s)
I20260809 16:39:42.772552 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:42.773530 2955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:42.776310 2955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f: Bootstrap starting.
I20260809 16:39:42.777015 2955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:42.777366 2955 log.cc:824] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:42.778268 2955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f: No bootstrap required, opened a new log
I20260809 16:39:42.780119 2955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } }
I20260809 16:39:42.780270 2955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:42.780292 2955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6770ec236a7a47d18d253b71e4f0696f, State: Initialized, Role: FOLLOWER
I20260809 16:39:42.780529 2955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } }
I20260809 16:39:42.780601 2955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:42.780624 2955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:42.780674 2955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:42.781363 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2876
I20260809 16:39:42.781381 2955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } }
I20260809 16:39:42.781512 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:42.781525 2955 leader_election.cc:302] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6770ec236a7a47d18d253b71e4f0696f; no voters:
I20260809 16:39:42.781723 2955 leader_election.cc:288] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:42.781807 2958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:42.781893 28922 external_mini_cluster.cc:1428] Setting key 3987d4f15f1a8ac40158e1201e75e3b0
I20260809 16:39:42.782016 2955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:42.782325 2958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [term 1 LEADER]: Becoming Leader. State: Replica: 6770ec236a7a47d18d253b71e4f0696f, State: Running, Role: LEADER
I20260809 16:39:42.782435 2958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } }
I20260809 16:39:42.782842 2958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6770ec236a7a47d18d253b71e4f0696f. Latest consensus state: current_term: 1 leader_uuid: "6770ec236a7a47d18d253b71e4f0696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } } }
I20260809 16:39:42.782976 2958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:42.783046 2959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6770ec236a7a47d18d253b71e4f0696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6770ec236a7a47d18d253b71e4f0696f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35575 } } }
I20260809 16:39:42.783136 2959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:42.784066 2971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:42.784618 2971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:42.786361 2971 catalog_manager.cc:1347] Generated new cluster ID: 372f058dbdc047feabd55897fac0cb60
I20260809 16:39:42.786428 2971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:39:42 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:42.794344 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:42.782442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.28.62.190:35575)
0809 16:39:42.782863 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:42.782874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:42.782909 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:39:42.783060 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:42.783068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:42.783144 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:42.783278 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:42.784372 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.785817 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.786963 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.787450 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.787604 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:42.790081 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:42.790086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:42.790087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:42.790106 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:42.792643 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:42.793075 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:42.793079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:42.793080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:42.793182 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:42.793407 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:42.793408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:42.793409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:42.793675 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:42.793870 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:42.793945 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 16:39:42.794118 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":80,"threads_started":1}
I20260809 16:39:42.811893 2971 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:42.812505 2971 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:42.815173 2971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6770ec236a7a47d18d253b71e4f0696f: Generated new TSK 0
I20260809 16:39:42.815369 2971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:42.852052 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:42.936924 2983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:42.937134 2983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:42.937165 2983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:42.938398 2983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:42.938444 2983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:42.938470 2983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:42.940114 2983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:42.940376 2983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:42.940626 2983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:42.945825 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:42.945827 2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:42.945825 2988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:42.945997 2983 server_base.cc:1034] running on GCE node
I20260809 16:39:42.946470 2983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:42.946766 2983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:42.947924 2983 hybrid_clock.cc:648] HybridClock initialized: now 1786293582947900 us; error 54 us; skew 500 ppm
Aug 09 16:39:42 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:42.950663 2983 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:42.951193 2983 webserver.cc:460] Webserver started at http://127.28.62.129:43463/ using document root <none> and password file <none>
I20260809 16:39:42.951474 2983 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:42.951550 2983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:42.951704 2983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:42.952816 2983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "71ec8b0ad93940c8829cbb8267edf22f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "717213236441fa4a5b614d2d766b3350"
server_key_iv: "9370c0312d517e2c5d7efa63cb50ff7e"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.953191 2983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "71ec8b0ad93940c8829cbb8267edf22f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "717213236441fa4a5b614d2d766b3350"
server_key_iv: "9370c0312d517e2c5d7efa63cb50ff7e"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.954581 2983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:42.955492 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:42.955672 2983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:42.955782 2983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "71ec8b0ad93940c8829cbb8267edf22f"
format_stamp: "Formatted at 2026-08-09 16:39:42 on dist-test-slave-wl2h"
server_key: "717213236441fa4a5b614d2d766b3350"
server_key_iv: "9370c0312d517e2c5d7efa63cb50ff7e"
server_key_version: "encryptionkey@0"
I20260809 16:39:42.955865 2983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:42.978645 2983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:42.981209 2983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:42.981418 2983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:42.981685 2983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:42.982131 2983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:42.982191 2983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:42.982234 2983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:42.982262 2983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:42.988521 2983 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:38371
I20260809 16:39:42.988574 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:38371 every 8 connection(s)
I20260809 16:39:42.988955 2983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:42 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:42.998247 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:42.989794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:37833 (local address 127.28.62.190:35575)
0809 16:39:42.989881 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:42.989883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:42.990053 (+ 170us) server_negotiation.cc:407] Connection header received
0809 16:39:42.991045 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:42.991046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:42.991061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:42.991120 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:42.991735 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.992283 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.993246 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.993440 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.993498 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:42.995643 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:42.995644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:42.995644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:42.995649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:42.997086 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:42.997443 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:42.997444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:42.997444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:42.997497 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:42.997711 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:42.997712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:42.997712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:42.997811 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:42.997862 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:42.998089 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 16:39:42.998181 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 16:39:42.998384 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:42.989753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35575 (local address 127.28.62.129:37833)
0809 16:39:42.989918 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:42.989962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:42.990912 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:42.991193 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:42.991197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:42.991464 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:42.991654 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:42.991659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.992373 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.992374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:42.993158 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:42.993160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:42.993522 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:42.993523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:42.993621 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:42.994102 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:42.994109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:42.995557 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:42.997157 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:42.997163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:42.997165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:42.997375 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:42.997557 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:42.997558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:42.997558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:42.997652 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:42.997867 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:42.997872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:42.997978 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 16:39:42.998058 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 16:39:42.998174 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260809 16:39:42.998759 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 2983
I20260809 16:39:42.998898 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:42.998976 3138 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35575
I20260809 16:39:42.999107 3138 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:42.999145 28922 external_mini_cluster.cc:1428] Setting key 5b5839094e6bd060714b67075c41197a
I20260809 16:39:42.999311 3138 heartbeater.cc:511] Master 127.28.62.190:35575 requested a full tablet report, sending...
I20260809 16:39:42.999858 2912 ts_manager.cc:194] Registered new tserver with Master: 71ec8b0ad93940c8829cbb8267edf22f (127.28.62.129:38371)
I20260809 16:39:43.000484 2912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:37833
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:43.026705 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:43.109848 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:43.110047 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:43.110071 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:43.111213 3145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:43.111258 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:43.111284 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:43.112797 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:43.112979 3145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:43.113148 3145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:43.118289 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:43.118314 3150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:43.118458 3145 server_base.cc:1034] running on GCE node
W20260809 16:39:43.118312 3153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:43.118752 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:43.118988 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:43.120189 3145 hybrid_clock.cc:648] HybridClock initialized: now 1786293583120170 us; error 39 us; skew 500 ppm
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.122814 3145 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:43.123320 3145 webserver.cc:460] Webserver started at http://127.28.62.130:45333/ using document root <none> and password file <none>
I20260809 16:39:43.123569 3145 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:43.123637 3145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:43.123790 3145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:43.124830 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "7d23f84167ba44d6a23a6c08b8c9f7dd"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "4fcf8f2c07b8e54b45a4a7d22ac82f03"
server_key_iv: "cc04aa772a3581ac4f6bb4f2a09abb64"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.125192 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "7d23f84167ba44d6a23a6c08b8c9f7dd"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "4fcf8f2c07b8e54b45a4a7d22ac82f03"
server_key_iv: "cc04aa772a3581ac4f6bb4f2a09abb64"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.126631 3145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:43.127492 3160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.127715 3145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:43.127808 3145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "7d23f84167ba44d6a23a6c08b8c9f7dd"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "4fcf8f2c07b8e54b45a4a7d22ac82f03"
server_key_iv: "cc04aa772a3581ac4f6bb4f2a09abb64"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.127892 3145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:43.155676 3145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:43.157814 3145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:43.157985 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:43.158272 3145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:43.158658 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:43.158726 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.158771 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:43.158798 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.164819 3145 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36217
I20260809 16:39:43.164873 3295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36217 every 8 connection(s)
I20260809 16:39:43.165186 3145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:43.172544 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3145
I20260809 16:39:43.172667 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:43.172920 28922 external_mini_cluster.cc:1428] Setting key 65e5a5062d92cf616f8e8df800e20529
I20260809 16:39:43.175570 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.165990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:49269 (local address 127.28.62.190:35575)
0809 16:39:43.166082 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.166084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.166174 (+ 90us) server_negotiation.cc:407] Connection header received
0809 16:39:43.167196 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.167198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.167209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.167268 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.167866 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.168370 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.169412 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.169592 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.169647 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.171912 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:43.171913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:43.171913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:43.171920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:43.173699 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.174373 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.174374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.174374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.174457 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.174786 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.174787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.174788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.174917 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:43.174997 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:43.175366 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 16:39:43.175479 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 16:39:43.175731 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.165903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35575 (local address 127.28.62.130:49269)
0809 16:39:43.166065 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:43.166107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:43.167086 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:43.167322 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:43.167325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:43.167590 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:43.167785 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:43.167789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.168464 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.168465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:43.169329 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:43.169331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.169697 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.169698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:43.169809 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.170402 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:43.170410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:43.171825 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:43.173810 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:43.173818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:43.173820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:43.174225 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:43.174520 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:43.174520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:43.174521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:43.174695 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:43.174987 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:43.174994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:43.175195 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 16:39:43.175303 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 16:39:43.175485 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260809 16:39:43.176219 3297 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35575
I20260809 16:39:43.176322 3297 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:43.176501 3297 heartbeater.cc:511] Master 127.28.62.190:35575 requested a full tablet report, sending...
I20260809 16:39:43.176904 2912 ts_manager.cc:194] Registered new tserver with Master: 7d23f84167ba44d6a23a6c08b8c9f7dd (127.28.62.130:36217)
I20260809 16:39:43.177397 2912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:49269
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:43.203097 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:43.286082 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:43.286293 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:43.286317 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:43.287521 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:43.287562 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:43.287596 3302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:43.289083 3302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:43.289318 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:43.289547 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:43.294718 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:43.294723 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:43.294776 3302 server_base.cc:1034] running on GCE node
W20260809 16:39:43.294910 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:43.295203 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:43.295449 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:43.296631 3302 hybrid_clock.cc:648] HybridClock initialized: now 1786293583296618 us; error 31 us; skew 500 ppm
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.299602 3302 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:43.300134 3302 webserver.cc:460] Webserver started at http://127.28.62.131:33495/ using document root <none> and password file <none>
I20260809 16:39:43.300386 3302 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:43.300490 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:43.300644 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:43.301813 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "ef2c192315464a3c963ebc71dc4ad408"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "bbd8c60943e112e535924573fd2bbcd0"
server_key_iv: "e1769e9e91a06a4a4ce7c8fdf5116afc"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.302234 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "ef2c192315464a3c963ebc71dc4ad408"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "bbd8c60943e112e535924573fd2bbcd0"
server_key_iv: "e1769e9e91a06a4a4ce7c8fdf5116afc"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.303817 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:43.304661 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.304911 3302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:43.305029 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "ef2c192315464a3c963ebc71dc4ad408"
format_stamp: "Formatted at 2026-08-09 16:39:43 on dist-test-slave-wl2h"
server_key: "bbd8c60943e112e535924573fd2bbcd0"
server_key_iv: "e1769e9e91a06a4a4ce7c8fdf5116afc"
server_key_version: "encryptionkey@0"
I20260809 16:39:43.305106 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:43.326354 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:43.328845 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:43.329015 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:43.329279 3302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:43.329608 3302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:43.329664 3302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.329710 3302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:43.329741 3302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:43.336022 3302 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36545
I20260809 16:39:43.336074 3453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36545 every 8 connection(s)
I20260809 16:39:43.336391 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:43.338735 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3302
I20260809 16:39:43.338843 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:43.339058 28922 external_mini_cluster.cc:1428] Setting key 91f2ec2369cb38cf1fb86f59d70196fa
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:43.346231 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.337140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:59015 (local address 127.28.62.190:35575)
0809 16:39:43.337186 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.337188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.337391 (+ 203us) server_negotiation.cc:407] Connection header received
0809 16:39:43.338321 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.338322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.338334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.338399 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.339142 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.339913 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.340959 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.341143 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.341197 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.343538 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:43.343539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:43.343540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:43.343547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:43.344826 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.345255 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.345256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.345256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.345321 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.345573 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.345574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.345574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.345693 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:43.345765 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:43.346045 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 16:39:43.346129 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:43.346429 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.337098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35575 (local address 127.28.62.131:59015)
0809 16:39:43.337258 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:43.337299 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:43.338217 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:43.338451 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:43.338454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:43.338717 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:43.338963 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:43.338968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.340021 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.340022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:43.340892 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:43.340895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.341222 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.341223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:43.341323 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.341885 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:43.341893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:43.343430 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:43.344913 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:43.344920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:43.344922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:43.345146 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:43.345393 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:43.345394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:43.345394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:43.345497 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:43.345766 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:43.345773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:43.345905 (+ 132us) client_negotiation.cc:769] Sending connection context
0809 16:39:43.346010 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 16:39:43.346167 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260809 16:39:43.346971 3454 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35575
I20260809 16:39:43.347090 3454 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:43.347327 3454 heartbeater.cc:511] Master 127.28.62.190:35575 requested a full tablet report, sending...
I20260809 16:39:43.347759 2912 ts_manager.cc:194] Registered new tserver with Master: ef2c192315464a3c963ebc71dc4ad408 (127.28.62.131:36545)
I20260809 16:39:43.348156 2912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:59015
I20260809 16:39:43.350905 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/test-admin.keytab.
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.368717 28922 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 16:39:43.370584 3463 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:43.369632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.28.62.129:38371)
0809 16:39:43.369812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.369818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.369835 (+ 17us) server_negotiation.cc:407] Connection header received
0809 16:39:43.369861 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.369863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.369877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.369964 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.370476 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48920: BlockingRecv error: recv got EOF from 127.0.0.1:48920 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
W20260809 16:39:43.370725 28922 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc' of type FILE
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.371735 28922 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.372627 28922 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:43.382545 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.373152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.28.62.129:38371)
0809 16:39:43.373204 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.373206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.373235 (+ 29us) server_negotiation.cc:407] Connection header received
0809 16:39:43.373304 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.373305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.373314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.373355 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.373783 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.374503 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.375314 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.375628 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.375693 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.378821 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:43.378829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:43.378829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:43.378836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:43.381082 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.381532 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.381535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.381535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.381616 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.381832 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.381833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.381833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.382022 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:43.382233 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:43.382330 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:39:43.382411 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260809 16:39:43.384852 3463 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:43.384219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.28.62.129:38371)
0809 16:39:43.384260 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.384263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.384306 (+ 43us) server_negotiation.cc:407] Connection header received
0809 16:39:43.384365 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.384366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.384377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.384442 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.384796 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48940: BlockingRecv error: recv got EOF from 127.0.0.1:48940 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260809 16:39:43.384975 28922 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc)
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.386004 28922 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.386952 28922 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:43.394028 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.387484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.28.62.129:38371)
0809 16:39:43.387536 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.387538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.387585 (+ 47us) server_negotiation.cc:407] Connection header received
0809 16:39:43.387659 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.387660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.387670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.387744 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.388118 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.388604 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.389292 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.389494 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.389569 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.391590 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:43.391591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:43.391591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:43.391598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:43.392963 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.393352 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.393353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.393353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.393403 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.393591 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.393592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.393592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.393676 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:43.393746 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:43.393857 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:39:43.393913 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260809 16:39:43.395561 3463 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:43.394978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.28.62.129:38371)
0809 16:39:43.395057 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.395059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.395160 (+ 101us) server_negotiation.cc:407] Connection header received
0809 16:39:43.395182 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.395183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.395210 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.395257 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.395519 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48960: BlockingRecv error: recv got EOF from 127.0.0.1:48960 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 16:39:43.395650 28922 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786293555794774-28922-0/krb5kdc/krb5cc' of type FILE
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.396657 28922 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:43.397485 28922 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:39:43.404090 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:43.397928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.28.62.129:38371)
0809 16:39:43.397987 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:43.397989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:43.397996 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:43.398071 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:43.398071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:43.398080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:43.398135 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:43.398506 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.398964 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.399667 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:43.399862 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:43.399936 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:43.401837 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:43.401839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:43.401839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:43.401846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:43.402980 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.403372 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.403374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.403374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.403432 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:43.403626 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:43.403627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:43.403627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:43.403722 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:43.403825 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:43.403931 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 16:39:43.403999 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:43.404731 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2983
I20260809 16:39:43.413915 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3145
I20260809 16:39:43.422412 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3302
I20260809 16:39:43.430785 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 2876
2026-08-09T16:39:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3087 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[3477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[3477](info): set up 2 sockets
Aug 09 16:39:43 dist-test-slave-wl2h krb5kdc[3477](info): commencing operation
krb5kdc: starting...
W20260809 16:39:44.585292 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:44 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab.
Entry for principal oryx/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab.
I20260809 16:39:44.711452 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35925
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab
--principal=oryx/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:44.792771 3491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:44.792979 3491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:44.793002 3491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:44.794067 3491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:44.794102 3491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:44.794116 3491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:44.794129 3491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:44.794142 3491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:44.795660 3491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab
--principal=oryx/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:44.795849 3491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:44.796023 3491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:44.801108 3496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:44.801105 3497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:44.801172 3499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:44.801404 3491 server_base.cc:1034] running on GCE node
I20260809 16:39:44.801662 3491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:44.801970 3491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:44.803148 3491 hybrid_clock.cc:648] HybridClock initialized: now 1786293584803134 us; error 37 us; skew 500 ppm
Aug 09 16:39:44 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:44.805706 3491 init.cc:377] Logged in from keytab as oryx/127.28.62.190@KRBTEST.COM (short username oryx)
I20260809 16:39:44.806093 3491 webserver.cc:460] Webserver started at http://127.28.62.190:42719/ using document root <none> and password file <none>
I20260809 16:39:44.806299 3491 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:44.806339 3491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:44.806427 3491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:44.807559 3491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "3e8887286f7849429b5e0e84d1b4184a"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "02e25de3c860abc22b54ed84bb912675"
server_key_iv: "37ad1a3feb5d953ace79c6548109cdab"
server_key_version: "encryptionkey@0"
I20260809 16:39:44.807931 3491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "3e8887286f7849429b5e0e84d1b4184a"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "02e25de3c860abc22b54ed84bb912675"
server_key_iv: "37ad1a3feb5d953ace79c6548109cdab"
server_key_version: "encryptionkey@0"
I20260809 16:39:44.809266 3491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:44.810084 3508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:44.810297 3491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:44.810415 3491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "3e8887286f7849429b5e0e84d1b4184a"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "02e25de3c860abc22b54ed84bb912675"
server_key_iv: "37ad1a3feb5d953ace79c6548109cdab"
server_key_version: "encryptionkey@0"
I20260809 16:39:44.810495 3491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:44.825412 3491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:44.827960 3491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:44.828131 3491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:44.832629 3491 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35925
I20260809 16:39:44.832666 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35925 every 8 connection(s)
I20260809 16:39:44.833033 3491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:44.833998 3573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:44.836321 3573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Bootstrap starting.
I20260809 16:39:44.836987 3573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:44.837069 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3491
I20260809 16:39:44.837204 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:44.837373 3573 log.cc:824] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:44.837448 28922 external_mini_cluster.cc:1428] Setting key 28c877c9e24a81e8017ec7ae91bb0c5f
I20260809 16:39:44.838274 3573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: No bootstrap required, opened a new log
I20260809 16:39:44.840793 3573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } }
I20260809 16:39:44.840965 3573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:44.841002 3573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8887286f7849429b5e0e84d1b4184a, State: Initialized, Role: FOLLOWER
I20260809 16:39:44.841115 3573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } }
I20260809 16:39:44.841202 3573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:44.841230 3573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:44.841279 3573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:44.841902 3573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } }
I20260809 16:39:44.841991 3573 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8887286f7849429b5e0e84d1b4184a; no voters:
I20260809 16:39:44.842159 3573 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:44.842190 3579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:44.842398 3579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 LEADER]: Becoming Leader. State: Replica: 3e8887286f7849429b5e0e84d1b4184a, State: Running, Role: LEADER
I20260809 16:39:44.842530 3573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:44.842540 3579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } }
I20260809 16:39:44.842995 3581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8887286f7849429b5e0e84d1b4184a. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } }
I20260809 16:39:44.843125 3581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:44.843376 3582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } }
I20260809 16:39:44.843492 3582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:44.843662 3589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:44.844187 3589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 16:39:44 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:44.845775 3589 catalog_manager.cc:1347] Generated new cluster ID: af1362a1d1a04665a414211f86a803cf
I20260809 16:39:44.845834 3589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:44.850178 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:44.838006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.28.62.190:35925)
0809 16:39:44.838183 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:44.838193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:44.838232 (+ 39us) server_negotiation.cc:407] Connection header received
0809 16:39:44.838717 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:44.838723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:44.838769 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:44.838878 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:44.839708 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:44.840559 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:44.841602 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:44.842070 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:44.842198 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:44.845156 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:44.845160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:44.845161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:44.845172 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:44.848530 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:44.848948 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:44.848951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:44.848952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:44.849052 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:44.849276 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:44.849277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:44.849277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:44.849565 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:44.849709 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:44.849869 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 16:39:44.849994 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260809 16:39:44.864703 3589 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:44.865259 3589 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:44.872345 3589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Generated new TSK 0
I20260809 16:39:44.872522 3589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
I20260809 16:39:44.904145 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:44.987605 3603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:44.987823 3603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:44.987847 3603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:44.989106 3603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:44.989158 3603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:44.989187 3603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:44.990676 3603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:44.990942 3603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:44.991227 3603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:44.996483 3611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:44.996495 3613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:44.996524 3603 server_base.cc:1034] running on GCE node
W20260809 16:39:44.996495 3610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:44.996887 3603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:44.997145 3603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:44.998318 3603 hybrid_clock.cc:648] HybridClock initialized: now 1786293584998311 us; error 33 us; skew 500 ppm
Aug 09 16:39:44 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.000932 3603 init.cc:377] Logged in from keytab as oryx/127.28.62.129@KRBTEST.COM (short username oryx)
I20260809 16:39:45.001435 3603 webserver.cc:460] Webserver started at http://127.28.62.129:38065/ using document root <none> and password file <none>
I20260809 16:39:45.001680 3603 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.001758 3603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.001912 3603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:45.002938 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "8c985aba99f8496d84710ebd3f5a3735"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "d6915a48aa420ae289c34f792974174f"
server_key_iv: "981e3da6fca55b5f08a5940565c57224"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.003371 3603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "8c985aba99f8496d84710ebd3f5a3735"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "d6915a48aa420ae289c34f792974174f"
server_key_iv: "981e3da6fca55b5f08a5940565c57224"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.004721 3603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:45.005585 3620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.005815 3603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:45.005926 3603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8c985aba99f8496d84710ebd3f5a3735"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "d6915a48aa420ae289c34f792974174f"
server_key_iv: "981e3da6fca55b5f08a5940565c57224"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.006016 3603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.020918 3603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.023094 3603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:45.023312 3603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.023582 3603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:45.023921 3603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:45.023977 3603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.024012 3603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:45.024055 3603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.030174 3603 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:37471
I20260809 16:39:45.030228 3754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:37471 every 8 connection(s)
I20260809 16:39:45.030536 3603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:45.039981 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.031268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:52955 (local address 127.28.62.190:35925)
0809 16:39:45.031333 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.031335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.031557 (+ 222us) server_negotiation.cc:407] Connection header received
0809 16:39:45.032109 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.032111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.032123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.032171 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.033134 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.033658 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.034665 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.034906 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.034985 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.037227 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.037228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.037229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.037236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.038521 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.038922 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.038923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.038923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.038988 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.039295 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.039296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.039297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.039420 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.039500 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.039778 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.039883 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 16:39:45.040126 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.031265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:52955)
0809 16:39:45.031424 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.031460 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.031980 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.032246 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.032249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.032816 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.033006 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.033011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.033764 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.033765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.034563 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.034565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.035008 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.035034 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.035164 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.035653 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.035661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.037099 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.038609 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.038617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.038619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.038839 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.039100 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.039101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.039102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.039220 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.039491 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.039498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.039634 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.039742 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.039899 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260809 16:39:45.040309 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3603
I20260809 16:39:45.040414 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:45.040660 28922 external_mini_cluster.cc:1428] Setting key fcbb7062806820c8a3e96553035e3d65
I20260809 16:39:45.040751 3755 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:45.040879 3755 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:45.041098 3755 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:45.041749 3530 ts_manager.cc:194] Registered new tserver with Master: 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471)
I20260809 16:39:45.042665 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:52955
WARNING: no policy specified for oryx/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
I20260809 16:39:45.069994 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:45.153354 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:45.153554 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:45.153578 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:45.154723 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:45.154763 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:45.154790 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:45.156177 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:45.156405 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:45.156658 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.161823 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.161809 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.161911 3764 server_base.cc:1034] running on GCE node
W20260809 16:39:45.161805 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.162302 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.162546 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.163713 3764 hybrid_clock.cc:648] HybridClock initialized: now 1786293585163703 us; error 40 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.166174 3764 init.cc:377] Logged in from keytab as oryx/127.28.62.130@KRBTEST.COM (short username oryx)
I20260809 16:39:45.166720 3764 webserver.cc:460] Webserver started at http://127.28.62.130:46085/ using document root <none> and password file <none>
I20260809 16:39:45.166978 3764 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.167071 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.167239 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:45.168299 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "57ba9869902c436e9d40f1df25fdebec"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "d4c07d5f4674d9bd5e5b6b69549dec95"
server_key_iv: "caf40d5db913097d38244406b7ad42f9"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.168665 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "57ba9869902c436e9d40f1df25fdebec"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "d4c07d5f4674d9bd5e5b6b69549dec95"
server_key_iv: "caf40d5db913097d38244406b7ad42f9"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.170022 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:45.170826 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.171056 3764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:45.171164 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "57ba9869902c436e9d40f1df25fdebec"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "d4c07d5f4674d9bd5e5b6b69549dec95"
server_key_iv: "caf40d5db913097d38244406b7ad42f9"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.171267 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.188901 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.190917 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:45.191184 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.191475 3764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:45.191807 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:45.191864 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.191908 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:45.191937 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.198048 3764 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:40813
I20260809 16:39:45.198128 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:40813 every 8 connection(s)
I20260809 16:39:45.198447 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:45.207218 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3764
I20260809 16:39:45.207352 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:45.207578 28922 external_mini_cluster.cc:1428] Setting key feea57756c5ef397747141437eb7c6bf
I20260809 16:39:45.208333 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.199319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:46691 (local address 127.28.62.190:35925)
0809 16:39:45.199414 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.199416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.199604 (+ 188us) server_negotiation.cc:407] Connection header received
0809 16:39:45.200172 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.200174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.200186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.200254 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.201198 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.201716 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.202703 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.202923 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.203031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.205103 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.205104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.205104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.205112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.206612 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.206991 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.206992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.206992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.207079 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.207498 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.207499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.207500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.207651 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.207730 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.208088 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.208173 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 16:39:45.208464 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.199289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:46691)
0809 16:39:45.199473 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.199520 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.200061 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.200322 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.200326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.200907 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.201095 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.201100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.201820 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.201821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.202610 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.202612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.203044 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.203045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.203173 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.203650 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.203658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.205009 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.206702 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.206708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.206709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.206912 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.207231 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.207232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.207232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.207377 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.207723 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.207729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.207892 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.208023 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.208190 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260809 16:39:45.209002 3915 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:45.209082 3915 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:45.209266 3915 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:45.209673 3530 ts_manager.cc:194] Registered new tserver with Master: 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
I20260809 16:39:45.210321 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:46691
WARNING: no policy specified for oryx/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
I20260809 16:39:45.238579 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:45.321796 3920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:45.322003 3920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:45.322028 3920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:45.323175 3920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:45.323221 3920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:45.323248 3920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:45.324753 3920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:45.324956 3920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:45.325145 3920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.330508 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.330483 3928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.330518 3920 server_base.cc:1034] running on GCE node
W20260809 16:39:45.330483 3926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.330989 3920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.331278 3920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.332458 3920 hybrid_clock.cc:648] HybridClock initialized: now 1786293585332444 us; error 28 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.335237 3920 init.cc:377] Logged in from keytab as oryx/127.28.62.131@KRBTEST.COM (short username oryx)
I20260809 16:39:45.335731 3920 webserver.cc:460] Webserver started at http://127.28.62.131:39325/ using document root <none> and password file <none>
I20260809 16:39:45.335938 3920 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.335978 3920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.336068 3920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:45.337074 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "a03d602c44f540a5a04aabcc2a423736"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "6dcf3bef600ab1d99f232e2714a802cc"
server_key_iv: "fb48b8c08f57ae051c983359719556f4"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.337370 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "a03d602c44f540a5a04aabcc2a423736"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "6dcf3bef600ab1d99f232e2714a802cc"
server_key_iv: "fb48b8c08f57ae051c983359719556f4"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.338713 3920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260809 16:39:45.339499 3936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.339663 3920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.339771 3920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "a03d602c44f540a5a04aabcc2a423736"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "6dcf3bef600ab1d99f232e2714a802cc"
server_key_iv: "fb48b8c08f57ae051c983359719556f4"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.339848 3920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.354954 3920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.357460 3920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:45.357632 3920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.357892 3920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:45.358291 3920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:45.358346 3920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.358382 3920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:45.358410 3920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.364627 3920 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36003
I20260809 16:39:45.364691 4074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36003 every 8 connection(s)
I20260809 16:39:45.364997 3920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:45.374133 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.365763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:34471 (local address 127.28.62.190:35925)
0809 16:39:45.365834 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.365836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.366022 (+ 186us) server_negotiation.cc:407] Connection header received
0809 16:39:45.366531 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.366532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.366541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.366597 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.367606 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.368106 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.369104 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.369347 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.369415 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.371542 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.371543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.371544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.371550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.372928 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.373309 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.373309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.373310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.373360 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.373567 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.373568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.373568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.373668 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.373729 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.373991 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.374068 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:39:45.374348 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 3920
I20260809 16:39:45.374406 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.365716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:34471)
0809 16:39:45.365883 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.365933 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.366449 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.366656 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.366659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.367324 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.367513 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.367517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.368210 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.368211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.369003 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.369005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.369434 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.369435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.369535 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.370006 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.370015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.371457 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.372999 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.373005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.373007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.373242 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.373418 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.373418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.373419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.373513 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.373728 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.373732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.373838 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.373919 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.374106 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260809 16:39:45.374447 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:45.374619 28922 external_mini_cluster.cc:1428] Setting key 47e511c54a209bf3b509040d3e8228e6
I20260809 16:39:45.375046 4075 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:45.375146 4075 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:45.375352 4075 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:45.375712 3530 ts_manager.cc:194] Registered new tserver with Master: a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:45.376045 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:34471
I20260809 16:39:45.386641 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.62.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.189.keytab.
Entry for principal oryx/127.28.62.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.189.keytab.
I20260809 16:39:45.413532 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:38341
--webserver_interface=127.28.62.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.189.keytab
--principal=oryx/127.28.62.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35925,127.28.62.189:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:45.495936 4081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:45.496145 4081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:45.496166 4081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:45.497298 4081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:45.497341 4081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:45.497356 4081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:45.497368 4081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:45.497411 4081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:45.498862 4081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35925,127.28.62.189:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.189.keytab
--principal=oryx/127.28.62.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.62.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
W20260809 16:39:45.498960 4081 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.62.190:35925,127.28.62.189:38341'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 16:39:45.499141 4081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:45.499389 4081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.504339 4088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.504334 4090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.504330 4086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.504465 4081 server_base.cc:1034] running on GCE node
I20260809 16:39:45.504784 4081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.505029 4081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.506172 4081 hybrid_clock.cc:648] HybridClock initialized: now 1786293585506149 us; error 38 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.509058 4081 init.cc:377] Logged in from keytab as oryx/127.28.62.189@KRBTEST.COM (short username oryx)
I20260809 16:39:45.509563 4081 webserver.cc:460] Webserver started at http://127.28.62.189:43721/ using document root <none> and password file <none>
I20260809 16:39:45.509790 4081 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.509836 4081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.509985 4081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:45.511257 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/instance:
uuid: "87a04e4b3a9b44faab8a5025613e2291"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "7a6de90153c77866f6e383d5219404c4"
server_key_iv: "251f47d70f70736dbf7883da57a6867c"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.511622 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal/instance:
uuid: "87a04e4b3a9b44faab8a5025613e2291"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "7a6de90153c77866f6e383d5219404c4"
server_key_iv: "251f47d70f70736dbf7883da57a6867c"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.513010 4081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:45.513800 4097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.514002 4081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:45.514122 4081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "87a04e4b3a9b44faab8a5025613e2291"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "7a6de90153c77866f6e383d5219404c4"
server_key_iv: "251f47d70f70736dbf7883da57a6867c"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.514194 4081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.548468 4081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.550688 4081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:45.550868 4081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.555142 4081 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.189:38341
I20260809 16:39:45.555182 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.189:38341 every 8 connection(s)
I20260809 16:39:45.555583 4081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
I20260809 16:39:45.556561 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:45.558108 4156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:39:45.559201 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 4081
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.189@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
I20260809 16:39:45.567979 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.558854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.28.62.190:35925)
0809 16:39:45.558929 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.558931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.559195 (+ 264us) server_negotiation.cc:407] Connection header received
0809 16:39:45.559717 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.559719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.559732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.559788 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.560567 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.561079 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.561996 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.562196 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.562289 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.564678 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.564679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.564680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.564687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.566313 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.566812 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.566813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.566813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.566865 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.567156 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.567157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.567157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.567372 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.567456 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.567799 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.567868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 16:39:45.568147 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.558818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42652)
0809 16:39:45.559054 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.559096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.559605 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.559932 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.559937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.560311 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.560465 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.560469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.561200 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.561201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.561899 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.561901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.562300 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.562301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.562420 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.562935 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.562945 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.564583 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.566405 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.566410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.566412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.566732 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.566932 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.566933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.566934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.567095 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.567456 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.567460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.567625 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.567743 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.567902 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":126,"threads_started":1}
I20260809 16:39:45.568548 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.559796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.28.62.189:38341)
0809 16:39:45.559967 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.559972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.560030 (+ 58us) server_negotiation.cc:407] Connection header received
0809 16:39:45.560065 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.560068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.560091 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.560183 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.560732 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.561537 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.562254 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.562585 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.562658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.565126 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.565134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.565134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.565141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.567039 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.567462 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.567465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.567465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.567524 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.567708 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.567709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.567710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.567861 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.568008 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.568353 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.568437 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260809 16:39:45.568820 4156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.189@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
I20260809 16:39:45.575055 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.569059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47302)
0809 16:39:45.569146 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.569202 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.569251 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.569375 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.569376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.569498 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.569522 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.569523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.570136 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.570136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.570597 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.570598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.570978 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.570979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.571075 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.571418 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.571422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.572627 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.574014 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.574015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.574015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.574181 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.574391 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.574392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.574392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.574442 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.574720 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.574720 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.574757 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.574825 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.574919 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 16:39:45.575220 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.569109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.28.62.189:38341)
0809 16:39:45.569143 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.569145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.569240 (+ 95us) server_negotiation.cc:407] Connection header received
0809 16:39:45.569310 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.569311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.569321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.569363 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.569595 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.570045 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.570683 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.570895 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.570979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.572725 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.572726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.572727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.572733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.573934 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.574266 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.574267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.574268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.574319 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.574511 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.574511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.574512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.574640 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.574728 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.574850 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.574967 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":16}
I20260809 16:39:45.576467 4156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Bootstrap starting.
I20260809 16:39:45.577239 4156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:45.577466 4156 log.cc:824] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:45.578191 4156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: No bootstrap required, opened a new log
I20260809 16:39:45.579727 4156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } }
I20260809 16:39:45.579852 4156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:45.579875 4156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a04e4b3a9b44faab8a5025613e2291, State: Initialized, Role: FOLLOWER
I20260809 16:39:45.579998 4156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } }
I20260809 16:39:45.580277 4156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:45.580265 4163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } } }
I20260809 16:39:45.580379 4163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 16:39:45.581925 4174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:39:45.581983 4174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:39:45.583681 4081 master_runner.cc:428] Detected that this master 87a04e4b3a9b44faab8a5025613e2291 is joining an existing cluster
I20260809 16:39:45.583788 4081 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.62.189:38341
I20260809 16:39:45.592159 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.584790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.28.62.190:35925)
0809 16:39:45.584842 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.584845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.585026 (+ 181us) server_negotiation.cc:407] Connection header received
0809 16:39:45.585067 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.585068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.585082 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.585165 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.585456 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.586249 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.586995 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.587320 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.587395 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.588568 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.588569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.588570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.588578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.590711 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.591218 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.591220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.591220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.591280 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.591511 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.591513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.591513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.591659 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.591752 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.591931 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.592020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:45.592175 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.584736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42660)
0809 16:39:45.584923 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.584933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.585015 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.585154 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.585155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.585312 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.585383 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.585385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.586336 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.586337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.586913 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.586914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.587467 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.587468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.587532 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.588029 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.588034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.588460 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.590806 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.590807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.590807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.591117 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.591372 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.591373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.591373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.591431 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.591750 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.591751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.591797 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.591885 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.592029 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260809 16:39:45.593763 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.584784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47312)
0809 16:39:45.587139 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.587150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.587241 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.587348 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.587349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.587514 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.587561 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.587562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.588328 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.588329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.588886 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.588888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.589405 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.589406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.589461 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.589891 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.589896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.590256 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.592131 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.592132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.592132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.592783 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.592998 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.592999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.592999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.593066 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.593360 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.593361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.593391 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.593462 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.593588 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2338,"thread_start_us":93,"threads_started":1}
I20260809 16:39:45.593842 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.584999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.28.62.189:38341)
0809 16:39:45.585039 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.585041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.587212 (+ 2171us) server_negotiation.cc:407] Connection header received
0809 16:39:45.587289 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.587290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.587299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.587347 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.587636 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.588210 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.588970 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.589283 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.589356 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.590361 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.590362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.590363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.590369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.592035 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.592857 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.592858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.592858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.592919 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.593141 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.593142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.593142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.593268 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.593313 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.593508 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.593670 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":24}
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.190@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
I20260809 16:39:45.602905 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.595666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.28.62.189:38341)
0809 16:39:45.595710 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.595711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.595862 (+ 151us) server_negotiation.cc:407] Connection header received
0809 16:39:45.595955 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.595956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.595966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.596018 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.596447 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.596889 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.597660 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.597848 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.597937 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.600189 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.600191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.600191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.600197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.601534 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.601978 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.601979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.601979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.602027 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.602299 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.602300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.602300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.602426 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.602491 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.602775 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.602844 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:45.602916 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.595594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47322)
0809 16:39:45.595740 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.595797 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.595888 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.596022 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.596025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.596309 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.596359 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.596361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.596973 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.596974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.597557 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.597558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.597940 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.597940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.598027 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.598482 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.598487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.600086 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.601620 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.601627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.601630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.601900 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.602095 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.602096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.602097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.602202 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.602505 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.602513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.602570 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.602663 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.602794 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260809 16:39:45.603708 3530 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.62.189:38341: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291"
member_type: NON_VOTER
last_known_addr {
host: "127.28.62.189"
port: 38341
}
attrs {
promote: true
}
}
dest_uuid: "3e8887286f7849429b5e0e84d1b4184a"
cas_config_opid_index: -1
I20260809 16:39:45.604051 3530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } }
I20260809 16:39:45.604624 4183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } }
I20260809 16:39:45.604933 4129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:45.605597 4183 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.62.189:38341
I20260809 16:39:45.605901 4129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Refusing update from remote peer 3e8887286f7849429b5e0e84d1b4184a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 16:39:45.605973 4081 master.cc:537] Master@127.28.62.189:38341 shutting down...
I20260809 16:39:45.605975 4184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } }
I20260809 16:39:45.606061 4184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:45.606215 4184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 16:39:45.607089 3511 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 16:39:45.607995 3511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a -> Peer 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189:38341): Couldn't send request to peer 87a04e4b3a9b44faab8a5025613e2291. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 16:39:45.608956 4081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:45.609066 4081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:45.609127 4081 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: stopping tablet replica
I20260809 16:39:45.621903 4081 master.cc:552] Master@127.28.62.189:38341 shutdown complete.
I20260809 16:39:45.621985 4081 master_runner.cc:305] Clearing existing system tablet
I20260809 16:39:45.622126 4081 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.622189 4081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.622850 4081 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260809 16:39:45.623407 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.623581 4081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.623660 4081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.623689 4081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "87a04e4b3a9b44faab8a5025613e2291"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "7a6de90153c77866f6e383d5219404c4"
server_key_iv: "251f47d70f70736dbf7883da57a6867c"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.624588 4081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:45.626451 4081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 16:39:45.626528 4081 log.cc:1196] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 16:39:45.626727 4081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Deleting consensus metadata
I20260809 16:39:45.626858 4081 master_runner.cc:315] Copying system tablet from 127.28.62.190:35925
I20260809 16:39:45.627508 4081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: tablet copy: Beginning tablet copy session from remote peer at address 127.28.62.190:35925
I20260809 16:39:45.634150 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.627828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42672)
0809 16:39:45.627958 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.627969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.628038 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.628197 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.628198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.628358 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.628412 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.628414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.629363 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.629364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.629908 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.629909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.630390 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.630391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.630442 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.630948 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.630952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.631339 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.633110 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.633111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.633111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.633286 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.633519 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.633519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.633520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.633584 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.633866 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.633867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.633907 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.633965 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.634055 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260809 16:39:45.634195 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.627908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.28.62.190:35925)
0809 16:39:45.627968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.627970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.628026 (+ 56us) server_negotiation.cc:407] Connection header received
0809 16:39:45.628110 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.628111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.628122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.628193 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.628483 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.629279 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.629995 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.630311 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.630385 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.631433 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.631435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.631435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.631443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.633022 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.633359 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.633360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.633360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.633422 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.633659 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.633660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.633660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.633797 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.633868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.634011 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.634092 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 16:39:45.634738 3553 tablet_copy_service.cc:140] P 3e8887286f7849429b5e0e84d1b4184a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 87a04e4b3a9b44faab8a5025613e2291 ({username='oryx', principal='oryx/127.28.62.189@KRBTEST.COM'} at 127.0.0.1:42672)
I20260809 16:39:45.634830 3553 tablet_copy_service.cc:161] P 3e8887286f7849429b5e0e84d1b4184a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 87a04e4b3a9b44faab8a5025613e2291 at {username='oryx', principal='oryx/127.28.62.189@KRBTEST.COM'} at 127.0.0.1:42672: session id = 87a04e4b3a9b44faab8a5025613e2291-00000000000000000000000000000000
I20260809 16:39:45.635536 3553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 16:39:45.636173 4081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 16:39:45.637480 4081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: tablet copy: Starting download of 0 data blocks...
I20260809 16:39:45.637645 4081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: tablet copy: Starting download of 1 WAL segments...
I20260809 16:39:45.638470 4081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 16:39:45.639351 3553 tablet_copy_service.cc:342] P 3e8887286f7849429b5e0e84d1b4184a: Request end of tablet copy session 87a04e4b3a9b44faab8a5025613e2291-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.62.189@KRBTEST.COM'} at 127.0.0.1:42672
I20260809 16:39:45.639400 3553 tablet_copy_service.cc:433] P 3e8887286f7849429b5e0e84d1b4184a: ending tablet copy session 87a04e4b3a9b44faab8a5025613e2291-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 87a04e4b3a9b44faab8a5025613e2291
I20260809 16:39:45.642402 4081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.644008 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.644027 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.644090 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.644124 4081 server_base.cc:1034] running on GCE node
I20260809 16:39:45.644250 4081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.644392 4081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.645522 4081 hybrid_clock.cc:648] HybridClock initialized: now 1786293585645509 us; error 28 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.646970 4081 init.cc:377] Logged in from keytab as oryx/127.28.62.189@KRBTEST.COM (short username oryx)
I20260809 16:39:45.647279 4081 webserver.cc:460] Webserver started at http://127.28.62.189:32937/ using document root <none> and password file <none>
I20260809 16:39:45.647408 4081 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.647457 4081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.648073 4081 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:45.648615 4211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.648777 4081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:45.648837 4081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "87a04e4b3a9b44faab8a5025613e2291"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "7a6de90153c77866f6e383d5219404c4"
server_key_iv: "251f47d70f70736dbf7883da57a6867c"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.648895 4081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.660073 4081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.661777 4081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.665071 4081 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.189:38341
I20260809 16:39:45.665445 4274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.189:38341 every 8 connection(s)
I20260809 16:39:45.665565 4081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
I20260809 16:39:45.666335 4275 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:39:45.666743 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Bootstrap starting.
I20260809 16:39:45.669123 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:39:45.669376 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: Bootstrap complete.
I20260809 16:39:45.669603 4275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } }
I20260809 16:39:45.669696 4275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87a04e4b3a9b44faab8a5025613e2291, State: Initialized, Role: LEARNER
I20260809 16:39:45.669739 4275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } }
I20260809 16:39:45.669902 4275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:45.671509 4291 catalog_manager.cc:1259] Loaded cluster ID: af1362a1d1a04665a414211f86a803cf
I20260809 16:39:45.671607 4291 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: loading cluster ID for follower catalog manager: success
I20260809 16:39:45.671317 4278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } }
I20260809 16:39:45.672490 4278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: LEARNER
I20260809 16:39:45.673338 4291 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: acquiring CA information for follower catalog manager: success
I20260809 16:39:45.673643 4291 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:39:45.677439 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.670235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.28.62.189:38341)
0809 16:39:45.670404 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.670406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.670418 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:45.670446 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.670447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.670461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.671449 (+ 988us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.671487 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.672462 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.673192 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.673385 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.673502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.674771 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.674772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.674773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.674780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.676279 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.676628 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.676629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.676629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.676687 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.676916 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.676917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.676917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.677032 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.677139 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.677251 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.677340 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.28.62.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.188.keytab.
Entry for principal oryx/127.28.62.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.188.keytab.
I20260809 16:39:45.706910 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:42983
--webserver_interface=127.28.62.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.188.keytab
--principal=oryx/127.28.62.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:45.791604 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:45.791867 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:45.791908 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:45.793097 4294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:45.793169 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:45.793193 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:45.793215 4294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:45.793246 4294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:45.794812 4294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.188.keytab
--principal=oryx/127.28.62.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:42983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.62.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:45.795135 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:45.795383 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.800293 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.800336 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.800379 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.800407 4294 server_base.cc:1034] running on GCE node
I20260809 16:39:45.800781 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.801084 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.802250 4294 hybrid_clock.cc:648] HybridClock initialized: now 1786293585802230 us; error 36 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.804924 4294 init.cc:377] Logged in from keytab as oryx/127.28.62.188@KRBTEST.COM (short username oryx)
I20260809 16:39:45.805393 4294 webserver.cc:460] Webserver started at http://127.28.62.188:39685/ using document root <none> and password file <none>
I20260809 16:39:45.805636 4294 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.805682 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.805855 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:45.806913 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/instance:
uuid: "3aa246cdeb6841d186c1f965a627f5d7"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "2d3cbadf199b62ff71013e9ea4b54841"
server_key_iv: "a409dd43abf6e7371fb1b4363febdb30"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.807358 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal/instance:
uuid: "3aa246cdeb6841d186c1f965a627f5d7"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "2d3cbadf199b62ff71013e9ea4b54841"
server_key_iv: "a409dd43abf6e7371fb1b4363febdb30"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.808808 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:45.809638 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.809816 4294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:45.809921 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "3aa246cdeb6841d186c1f965a627f5d7"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "2d3cbadf199b62ff71013e9ea4b54841"
server_key_iv: "a409dd43abf6e7371fb1b4363febdb30"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.809998 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.827538 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.829438 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:45.829604 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.834080 4294 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.188:42983
I20260809 16:39:45.834146 4370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.188:42983 every 8 connection(s)
I20260809 16:39:45.834515 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
I20260809 16:39:45.835619 4371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:45.837262 4371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:39:45.842978 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 4294
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.188@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:45.847069 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.838074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.28.62.190:35925)
0809 16:39:45.838155 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.838157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.838315 (+ 158us) server_negotiation.cc:407] Connection header received
0809 16:39:45.838817 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.838819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.838831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.838888 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.839568 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.840100 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.841086 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.841295 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.841383 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.843991 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.843993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.843993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.844008 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.845565 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.845953 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.845954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.845954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.846020 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.846298 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.846299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.846299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.846410 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.846463 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.846813 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.846905 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 16:39:45.847163 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.837954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42674)
0809 16:39:45.838171 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.838215 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.838703 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.838961 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.838964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.839282 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.839470 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.839474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.840202 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.840204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.840992 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.840994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.841406 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.841407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.841541 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.842078 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.842086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.843892 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.845641 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.845646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.845648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.845879 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.846088 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.846089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.846089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.846229 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.846475 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.846480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.846624 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.846739 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.846892 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260809 16:39:45.847949 4371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.188@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
I20260809 16:39:45.853701 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.843562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.28.62.188:42983)
0809 16:39:45.843725 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.843730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.843749 (+ 19us) server_negotiation.cc:407] Connection header received
0809 16:39:45.843771 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.843773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.843793 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.843878 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.844502 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.845294 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.846183 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.846603 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.846673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.850278 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.850286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.850286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.850292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.852288 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.852683 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.852685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.852685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.852738 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.853112 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.853112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.853113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.853261 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.853366 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.853492 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.853573 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260809 16:39:45.854996 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.848234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47348)
0809 16:39:45.848290 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.848302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.848349 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.848509 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.848509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.848624 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.848673 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.848674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.849653 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.849654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.850197 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.850199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.850556 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.850557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.850613 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.850950 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.850954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.852379 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.854186 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.854186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.854187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.854421 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.854594 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.854594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.854595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.854638 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.854804 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.854804 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.854833 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.854876 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.854939 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 16:39:45.855079 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.848326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.28.62.189:38341)
0809 16:39:45.848375 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.848376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.848387 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:45.848411 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.848411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.848427 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.848504 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.848768 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.849560 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.850279 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.850491 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.850562 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.852446 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.852447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.852447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.852455 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.854113 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.854484 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.854485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.854485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.854538 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.854672 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.854672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.854673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.854768 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.854813 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.854893 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.854959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:45.855491 4371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.188@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
I20260809 16:39:45.861085 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.855751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42944)
0809 16:39:45.855806 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.855816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.855850 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.856041 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.856041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.856144 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.856170 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.856171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.856718 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.856719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.857159 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.857160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.857469 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.857469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.857503 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.857907 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.857912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.858984 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.860299 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.860300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.860300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.860450 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.860619 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.860619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.860620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.860667 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.860880 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.860880 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.860911 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.860958 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.861029 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 16:39:45.861148 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.855922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.28.62.188:42983)
0809 16:39:45.855955 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.855956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.855963 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:45.855986 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.855987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.855995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.856039 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.856225 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.856654 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.857225 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.857408 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.857456 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.859108 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.859109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.859109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.859115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.860225 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.860516 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.860517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.860517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.860563 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.860727 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.860728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.860728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.860815 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.860862 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.860981 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.861073 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":16}
I20260809 16:39:45.862464 4371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Bootstrap starting.
I20260809 16:39:45.863358 4371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:45.863637 4371 log.cc:824] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:45.864312 4371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: No bootstrap required, opened a new log
I20260809 16:39:45.865732 4371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } }
I20260809 16:39:45.865882 4371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:45.865933 4371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa246cdeb6841d186c1f965a627f5d7, State: Initialized, Role: FOLLOWER
I20260809 16:39:45.866077 4371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } }
I20260809 16:39:45.866372 4371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:45.866336 4380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } } }
I20260809 16:39:45.866444 4380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 16:39:45.868182 4392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:39:45.868245 4392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:39:45.870501 4294 master_runner.cc:428] Detected that this master 3aa246cdeb6841d186c1f965a627f5d7 is joining an existing cluster
I20260809 16:39:45.870551 4294 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.62.188:42983
I20260809 16:39:45.878623 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.28.62.189:38341)
0809 16:39:45.871689 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.871692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.871988 (+ 296us) server_negotiation.cc:407] Connection header received
0809 16:39:45.872025 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.872026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.872036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.872106 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.872566 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873366 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.874687 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.874970 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.875030 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.876072 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.876073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.876074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.876081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.877442 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.877797 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.877798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.877798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.877854 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.878062 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.878063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.878063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.878174 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.878249 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.878422 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.878498 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:45.878661 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47362)
0809 16:39:45.871913 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.871922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.871979 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.872164 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.872165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.872406 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.872478 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.872481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873515 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.873515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.874607 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.874609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.875098 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.875099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.875164 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.875637 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.875640 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.875964 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.877527 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.877528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.877528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.877706 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.877925 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.877926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.877926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.877984 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.878253 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.878254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.878311 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.878396 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.878502 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":23,"thread_start_us":116,"threads_started":1}
I20260809 16:39:45.903571 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.28.62.188:42983)
0809 16:39:45.871692 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.871694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.871967 (+ 273us) server_negotiation.cc:407] Connection header received
0809 16:39:45.872070 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.872071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.872079 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.872110 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.872875 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873596 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.874631 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.874927 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.874969 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.900546 (+ 25577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.900548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.900548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.900554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.902240 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902729 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.902730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.902731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.902809 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.903006 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.903031 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.903031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.903171 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.903214 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.903387 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.903468 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:39:45.903687 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.28.62.190:35925)
0809 16:39:45.871743 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.871745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.872127 (+ 382us) server_negotiation.cc:407] Connection header received
0809 16:39:45.872232 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.872233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.872244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.872321 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.872693 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873421 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.874608 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.874913 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.874952 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.900547 (+ 25595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.900549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.900549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.900561 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.902283 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902829 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.902830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.902831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.902894 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.903109 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.903110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.903110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.903237 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.903309 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.903495 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.903578 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 16:39:45.903712 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42690)
0809 16:39:45.872075 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.872084 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.872150 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.872324 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.872324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.872509 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.872599 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.872601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873515 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.873515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.874507 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.874510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.875055 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.875056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.875123 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.899707 (+ 24584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.899714 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.900391 (+ 677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.902399 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.902401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.902692 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.902949 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.902950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.903046 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.903312 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.903313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.903378 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.903449 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.903570 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":78,"thread_start_us":69,"threads_started":1}
I20260809 16:39:45.904491 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.871710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42954)
0809 16:39:45.871901 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.871912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.871987 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.872169 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.872169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.872724 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.872785 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.872786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.873721 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.873721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.874542 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.874545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.875099 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.875100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.875160 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.899855 (+ 24695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.899860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.900391 (+ 531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.902348 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.902350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.902651 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.902869 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.902870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.902870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.902933 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.903254 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.903255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.903290 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.904256 (+ 966us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.904364 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260809 16:39:45.905090 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3603
I20260809 16:39:45.914927 4380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:45.915066 4380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } }
I20260809 16:39:45.915326 4380 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925), 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189:38341)
I20260809 16:39:45.915844 3540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa246cdeb6841d186c1f965a627f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8887286f7849429b5e0e84d1b4184a" is_pre_election: true
I20260809 16:39:45.916204 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa246cdeb6841d186c1f965a627f5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a04e4b3a9b44faab8a5025613e2291" is_pre_election: true
I20260809 16:39:45.916307 4245 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Handling vote request from an unknown peer 3aa246cdeb6841d186c1f965a627f5d7
I20260809 16:39:45.916347 4245 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 3aa246cdeb6841d186c1f965a627f5d7 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260809 16:39:45.916747 4313 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aa246cdeb6841d186c1f965a627f5d7; no voters: 3e8887286f7849429b5e0e84d1b4184a, 87a04e4b3a9b44faab8a5025613e2291
I20260809 16:39:45.916992 4380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:45.917800 4380 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 16:39:45.918630 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:37471
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:35925
--webserver_port=38065
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293584, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.190@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
I20260809 16:39:45.923178 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.905972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.28.62.188:42983)
0809 16:39:45.914195 (+ 8223us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.914197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.914204 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:45.914236 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.914237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.914246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.914327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.914925 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.915651 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.916593 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.916869 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.916932 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.920409 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.920411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.920412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.920421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.921929 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.922313 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.922315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.922315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.922371 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.922598 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.922598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.922599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.922727 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.922805 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.922968 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.923082 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8199}
I20260809 16:39:45.923188 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.905872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42964)
0809 16:39:45.913773 (+ 7901us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.913788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.913851 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.914337 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.914338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.914598 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.914674 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.914678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.915737 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.915738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.916500 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.916504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.917957 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.917959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.918026 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.918583 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.918588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.920292 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.922043 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.922044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.922045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.922235 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.922459 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.922460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.922461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.922524 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.922799 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.922800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.922851 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.922941 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.923089 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7879}
I20260809 16:39:45.924023 3530 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.62.188:42983: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7"
member_type: NON_VOTER
last_known_addr {
host: "127.28.62.188"
port: 42983
}
attrs {
promote: true
}
}
dest_uuid: "3e8887286f7849429b5e0e84d1b4184a"
cas_config_opid_index: 5
I20260809 16:39:45.924304 3530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } }
I20260809 16:39:45.924655 4183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.925451 4183 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.62.188:42983
I20260809 16:39:45.925527 4183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.925601 4183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:45.926126 4294 master.cc:537] Master@127.28.62.188:42983 shutting down...
W20260809 16:39:45.926831 3512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a -> Peer 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188:42983): Couldn't send request to peer 3aa246cdeb6841d186c1f965a627f5d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 16:39:45.929168 4294 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:45.929256 4294 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:45.929296 4294 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: stopping tablet replica
I20260809 16:39:45.930424 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.925880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47366)
0809 16:39:45.925998 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.926009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.926056 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.926341 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.926343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.926345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:45.926390 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.926392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.927705 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.927706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.928763 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.928767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.930113 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.930114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.930183 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.930225 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.930283 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.930338 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 16:39:45.930697 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.925921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.28.62.189:38341)
0809 16:39:45.925968 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.925970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.926108 (+ 138us) server_negotiation.cc:407] Connection header received
0809 16:39:45.926151 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.926152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.926178 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.926238 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:45.926477 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.927366 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.928873 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.929907 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.929985 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.930076 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.930572 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.930631 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:45.930912 4245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Refusing update from remote peer 3e8887286f7849429b5e0e84d1b4184a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 16:39:45.931190 4184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 16:39:45.931803 4245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.933104 4278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8887286f7849429b5e0e84d1b4184a. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.933213 4278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: LEARNER
I20260809 16:39:45.933182 4294 master.cc:552] Master@127.28.62.188:42983 shutdown complete.
I20260809 16:39:45.933259 4278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: NON_VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: true } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.933315 4278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: LEARNER
I20260809 16:39:45.933383 4294 master_runner.cc:305] Clearing existing system tablet
I20260809 16:39:45.933607 4294 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.933718 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.934032 4184 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: attempting to promote NON_VOTER 87a04e4b3a9b44faab8a5025613e2291 to VOTER
I20260809 16:39:45.934273 4184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } }
I20260809 16:39:45.934842 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.934719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42968)
0809 16:39:45.934765 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.934810 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 16:39:45.934878 4245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 LEARNER]: Refusing update from remote peer 3e8887286f7849429b5e0e84d1b4184a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 16:39:45.935075 4184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 16:39:45.935153 4294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:45.935292 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.935204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42974)
0809 16:39:45.935243 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.935266 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260809 16:39:45.935352 3512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a -> Peer 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188:42983): Couldn't send request to peer 3aa246cdeb6841d186c1f965a627f5d7. Status: Network error: Client connection negotiation failed: client connection to 127.28.62.188:42983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 16:39:45.935762 4183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.936551 4183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.936652 4183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:45.936569 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.936815 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:45.936952 4245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.937652 4404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.937803 4404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:39:45.939188 4407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.939380 4294 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.939435 4294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260809 16:39:45.939464 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "3aa246cdeb6841d186c1f965a627f5d7"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "2d3cbadf199b62ff71013e9ea4b54841"
server_key_iv: "a409dd43abf6e7371fb1b4363febdb30"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.940111 4294 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:45.941449 4294 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 16:39:45.941533 4294 log.cc:1196] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 16:39:45.941743 4294 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Deleting consensus metadata
I20260809 16:39:45.941886 4294 master_runner.cc:315] Copying system tablet from 127.28.62.190:35925
I20260809 16:39:45.942581 4294 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: tablet copy: Beginning tablet copy session from remote peer at address 127.28.62.190:35925
I20260809 16:39:45.949318 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.942933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:42694)
0809 16:39:45.943127 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:45.943136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:45.943212 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:45.943412 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:45.943413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:45.943578 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:45.943630 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.943632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.944414 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.944415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.944958 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:45.944960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.945347 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.945348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:45.945422 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.945928 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:45.945934 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:45.946254 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:45.948038 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.948039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.948040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.948320 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.948547 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:45.948548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:45.948548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:45.948615 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:45.948941 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:45.948942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:45.948992 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 16:39:45.949061 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 16:39:45.949170 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260809 16:39:45.949406 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:45.942964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.28.62.190:35925)
0809 16:39:45.943032 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:45.943035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:45.943200 (+ 165us) server_negotiation.cc:407] Connection header received
0809 16:39:45.943277 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:45.943278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:45.943288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:45.943347 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:45.943715 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.944327 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.945050 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:45.945270 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:45.945348 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:45.946369 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:45.946371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:45.946371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:45.946378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:45.947932 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.948397 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.948398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.948398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.948467 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:45.948687 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:45.948688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:45.948688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:45.948873 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:45.948950 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:45.949087 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 16:39:45.949169 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:45.949972 3553 tablet_copy_service.cc:140] P 3e8887286f7849429b5e0e84d1b4184a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3aa246cdeb6841d186c1f965a627f5d7 ({username='oryx', principal='oryx/127.28.62.188@KRBTEST.COM'} at 127.0.0.1:42694)
I20260809 16:39:45.950063 3553 tablet_copy_service.cc:161] P 3e8887286f7849429b5e0e84d1b4184a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3aa246cdeb6841d186c1f965a627f5d7 at {username='oryx', principal='oryx/127.28.62.188@KRBTEST.COM'} at 127.0.0.1:42694: session id = 3aa246cdeb6841d186c1f965a627f5d7-00000000000000000000000000000000
I20260809 16:39:45.950543 3553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 16:39:45.951092 4294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 16:39:45.952081 4294 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: tablet copy: Starting download of 0 data blocks...
I20260809 16:39:45.952260 4294 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: tablet copy: Starting download of 1 WAL segments...
I20260809 16:39:45.952978 4294 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 16:39:45.953814 3553 tablet_copy_service.cc:342] P 3e8887286f7849429b5e0e84d1b4184a: Request end of tablet copy session 3aa246cdeb6841d186c1f965a627f5d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.62.188@KRBTEST.COM'} at 127.0.0.1:42694
I20260809 16:39:45.953890 3553 tablet_copy_service.cc:433] P 3e8887286f7849429b5e0e84d1b4184a: ending tablet copy session 3aa246cdeb6841d186c1f965a627f5d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3aa246cdeb6841d186c1f965a627f5d7
I20260809 16:39:45.956656 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:45.958242 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.958451 4416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:45.958546 4418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:45.958776 4294 server_base.cc:1034] running on GCE node
I20260809 16:39:45.958902 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:45.959075 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:45.960206 4294 hybrid_clock.cc:648] HybridClock initialized: now 1786293585960197 us; error 27 us; skew 500 ppm
Aug 09 16:39:45 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293585, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:45.961644 4294 init.cc:377] Logged in from keytab as oryx/127.28.62.188@KRBTEST.COM (short username oryx)
I20260809 16:39:45.961959 4294 webserver.cc:460] Webserver started at http://127.28.62.188:42487/ using document root <none> and password file <none>
I20260809 16:39:45.962085 4294 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:45.962123 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:45.962796 4294 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:45.963469 4426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.963642 4294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:45.963687 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "3aa246cdeb6841d186c1f965a627f5d7"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "2d3cbadf199b62ff71013e9ea4b54841"
server_key_iv: "a409dd43abf6e7371fb1b4363febdb30"
server_key_version: "encryptionkey@0"
I20260809 16:39:45.963754 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:45.985076 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:45.986920 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:45.990456 4294 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.188:42983
I20260809 16:39:45.990796 4483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.188:42983 every 8 connection(s)
I20260809 16:39:45.990883 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
I20260809 16:39:45.991761 4484 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:39:45.992219 4484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Bootstrap starting.
I20260809 16:39:45.994573 4484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:39:45.994819 4484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Bootstrap complete.
I20260809 16:39:45.995085 4484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } }
I20260809 16:39:45.995188 4484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3aa246cdeb6841d186c1f965a627f5d7, State: Initialized, Role: LEARNER
I20260809 16:39:45.995244 4484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } }
I20260809 16:39:45.995433 4484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:45.996824 4487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:45.996939 4487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: LEARNER
I20260809 16:39:45.997097 4499 catalog_manager.cc:1259] Loaded cluster ID: af1362a1d1a04665a414211f86a803cf
I20260809 16:39:45.997175 4499 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: loading cluster ID for follower catalog manager: success
I20260809 16:39:45.998369 4499 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: acquiring CA information for follower catalog manager: success
I20260809 16:39:45.998647 4499 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 16:39:46.042445 4402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:46.042747 4402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:46.042794 4402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:46.044602 4402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:46.044684 4402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:46.044739 4402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:46.046994 4402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:37471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=38065
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:46.047317 4402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:46.047606 4402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:46.053351 4507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:46.053440 4504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:46.053705 4505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:46.053798 4402 server_base.cc:1034] running on GCE node
I20260809 16:39:46.054059 4402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:46.054311 4402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:46.055473 4402 hybrid_clock.cc:648] HybridClock initialized: now 1786293586055459 us; error 32 us; skew 500 ppm
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:46.057997 4402 init.cc:377] Logged in from keytab as oryx/127.28.62.129@KRBTEST.COM (short username oryx)
I20260809 16:39:46.058465 4402 webserver.cc:460] Webserver started at http://127.28.62.129:38065/ using document root <none> and password file <none>
I20260809 16:39:46.058710 4402 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:46.058768 4402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:46.060177 4402 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:46.060789 4518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.060949 4402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:46.061041 4402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8c985aba99f8496d84710ebd3f5a3735"
format_stamp: "Formatted at 2026-08-09 16:39:44 on dist-test-slave-wl2h"
server_key: "d6915a48aa420ae289c34f792974174f"
server_key_iv: "981e3da6fca55b5f08a5940565c57224"
server_key_version: "encryptionkey@0"
I20260809 16:39:46.061391 4402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:46.071892 4402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:46.073871 4402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:46.074016 4402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:46.074224 4402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:46.074592 4402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:46.074633 4402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.074656 4402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:46.074668 4402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.080495 4402 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:37471
I20260809 16:39:46.080572 4646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:37471 every 8 connection(s)
I20260809 16:39:46.080915 4402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:46.085894 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 4402
I20260809 16:39:46.086079 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3764
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
I20260809 16:39:46.093848 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:52087 (local address 127.28.62.189:38341)
0809 16:39:46.081964 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.081966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.081977 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:46.082577 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.082578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.082591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.082650 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.083595 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084359 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085526 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.085838 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085914 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.089650 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.089651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.089651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.089660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.091986 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092652 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.092654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.092654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.092725 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.093042 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.093043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.093043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.093193 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.093237 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.093642 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.093749 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:46.093848 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:52047 (local address 127.28.62.190:35925)
0809 16:39:46.081741 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.081744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.082120 (+ 376us) server_negotiation.cc:407] Connection header received
0809 16:39:46.082732 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.082733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.082746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.082811 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.083597 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084354 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085971 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.086686 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.086746 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.089650 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.089651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.089652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.089660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.091490 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092652 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.092654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.092654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.092716 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.093043 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.093043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.093044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.093182 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.093237 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.093646 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.093745 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:46.094550 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.28.62.129:52087)
0809 16:39:46.081830 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.081878 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.082472 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.082722 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.082726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.083249 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.083487 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.083492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084457 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.084458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.085445 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.085447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.085917 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.085993 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.087807 (+ 1814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.087812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.089531 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.092110 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.092122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.092555 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.092799 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.092800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.092957 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.093289 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.093297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.093480 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.094113 (+ 633us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.094340 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":200,"thread_start_us":102,"threads_started":1}
I20260809 16:39:46.094550 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:52047)
0809 16:39:46.082043 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.082052 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.082486 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.082840 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.082841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.083234 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.083487 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.083492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084457 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.084458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.085440 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.085442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.086798 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.086799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.086869 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.087281 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.087290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.089531 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.092154 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.092155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.092555 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.092799 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.092800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.092800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.092957 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.093289 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.093297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.093480 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.094006 (+ 526us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.094290 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":111,"thread_start_us":97,"threads_started":1}
I20260809 16:39:46.104369 4649 heartbeater.cc:347] Connected to a master server at 127.28.62.189:38341
I20260809 16:39:46.104496 4648 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:46.104540 4648 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.104503 4649 heartbeater.cc:464] Registering TS with master...
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
I20260809 16:39:46.104787 4648 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:46.105196 3530 ts_manager.cc:194] Re-registered known tserver with Master: 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471)
I20260809 16:39:46.104787 4649 heartbeater.cc:511] Master 127.28.62.189:38341 requested a full tablet report, sending...
I20260809 16:39:46.105720 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:52047
I20260809 16:39:46.105719 4233 ts_manager.cc:194] Registered new tserver with Master: 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471)
I20260809 16:39:46.109004 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:40813
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:35925
--webserver_port=46085
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:39:46.109179 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:38817 (local address 127.28.62.188:42983)
0809 16:39:46.082161 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.082164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.082172 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:46.082634 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.082636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.082651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.082713 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.083600 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084244 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085373 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.085647 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085721 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.105378 (+ 19657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.105380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.105381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.105390 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.107969 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.108331 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.108333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.108334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.108401 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.108604 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.108605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.108605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.108740 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.108830 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.108982 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.109080 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":59,"threads_started":1}
I20260809 16:39:46.109189 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.081655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.28.62.129:38817)
0809 16:39:46.082044 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.082053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.082480 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.083183 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.083184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.083336 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.083486 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.083490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.084363 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.084364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.085277 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.085279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.085734 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.085735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.085873 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.103607 (+ 17734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.103612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.105201 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.108062 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.108063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.108063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.108237 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.108471 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.108472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.108472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.108523 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.108819 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.108819 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.108867 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.108953 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.109077 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":290,"thread_start_us":61,"threads_started":1}
I20260809 16:39:46.109618 4647 heartbeater.cc:347] Connected to a master server at 127.28.62.188:42983
I20260809 16:39:46.109671 4647 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.109767 4647 heartbeater.cc:511] Master 127.28.62.188:42983 requested a full tablet report, sending...
I20260809 16:39:46.110191 4445 ts_manager.cc:194] Registered new tserver with Master: 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471)
W20260809 16:39:46.197190 4657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:46.197404 4657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:46.197428 4657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:46.198681 4657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:46.198729 4657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:46.198755 4657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:46.200294 4657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=46085
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:46.200506 4657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:46.200686 4657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:46.205848 4662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:46.205879 4666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:46.206118 4657 server_base.cc:1034] running on GCE node
W20260809 16:39:46.205883 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:46.206393 4657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:46.206652 4657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:46.207830 4657 hybrid_clock.cc:648] HybridClock initialized: now 1786293586207817 us; error 27 us; skew 500 ppm
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:46.210368 4657 init.cc:377] Logged in from keytab as oryx/127.28.62.130@KRBTEST.COM (short username oryx)
I20260809 16:39:46.210846 4657 webserver.cc:460] Webserver started at http://127.28.62.130:46085/ using document root <none> and password file <none>
I20260809 16:39:46.211097 4657 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:46.211159 4657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:46.212589 4657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:46.213213 4675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.213421 4657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:46.213500 4657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "57ba9869902c436e9d40f1df25fdebec"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "d4c07d5f4674d9bd5e5b6b69549dec95"
server_key_iv: "caf40d5db913097d38244406b7ad42f9"
server_key_version: "encryptionkey@0"
I20260809 16:39:46.213863 4657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:46.220474 4657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:46.222342 4657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:46.222492 4657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:46.222734 4657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:46.223112 4657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:46.223168 4657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.223219 4657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:46.223246 4657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.229349 4657 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:40813
I20260809 16:39:46.229589 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:40813 every 8 connection(s)
I20260809 16:39:46.229806 4657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:46.234970 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 4657
I20260809 16:39:46.235164 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3920
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
I20260809 16:39:46.251870 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:36003
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:35925
--webserver_port=39325
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:39:46.252476 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:49419 (local address 127.28.62.189:38341)
0809 16:39:46.230907 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.230909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.230925 (+ 16us) server_negotiation.cc:407] Connection header received
0809 16:39:46.232230 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.232231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.232244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.232312 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.233950 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.234416 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.240263 (+ 5847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.240576 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.240639 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.248530 (+ 7891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.248532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.248532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.248541 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.250274 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.250848 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.250850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.250850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.250913 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.251583 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.251584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.251585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.251749 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.251810 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.252262 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.252364 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:39:46.252735 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:41707 (local address 127.28.62.190:35925)
0809 16:39:46.230723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.230725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.232204 (+ 1479us) server_negotiation.cc:407] Connection header received
0809 16:39:46.232235 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.232236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.232248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.232303 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.233034 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.233838 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.245758 (+ 11920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.246040 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.246120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.248915 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.248917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.248917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.248925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.250797 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.251534 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.251535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.251536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.251596 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.252102 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.252103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.252103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.252251 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.252303 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.252493 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.252582 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:46.253106 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.28.62.130:49419)
0809 16:39:46.230780 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.230854 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.231925 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.232330 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.232331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.232766 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.232935 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.232941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.234501 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.234502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.240136 (+ 5634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.240141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.240712 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.240714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.240865 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.245825 (+ 4960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.245837 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.248436 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.250379 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.250387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.250390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.250756 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.250993 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.250994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.250994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.251450 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.251854 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.251860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.252087 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.252606 (+ 519us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.252808 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260809 16:39:46.253116 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:41707)
0809 16:39:46.232116 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.232126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.232178 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.232356 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.232357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.232736 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.232933 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.232938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.233964 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.233964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.245626 (+ 11662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.245630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.246188 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.246189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.246255 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.246789 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.246794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.248817 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.251217 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.251218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.251218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.251446 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.251673 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.251673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.251673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.251748 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.252340 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.252341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.252383 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.252810 (+ 427us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.252933 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"mutex_wait_us":361,"thread_start_us":84,"threads_started":1}
I20260809 16:39:46.253582 4807 heartbeater.cc:347] Connected to a master server at 127.28.62.189:38341
I20260809 16:39:46.253677 4807 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.253893 4807 heartbeater.cc:511] Master 127.28.62.189:38341 requested a full tablet report, sending...
I20260809 16:39:46.254005 4806 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:46.254068 4806 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.254144 4806 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:46.254362 3530 ts_manager.cc:194] Re-registered known tserver with Master: 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
I20260809 16:39:46.254364 4233 ts_manager.cc:194] Registered new tserver with Master: 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
I20260809 16:39:46.254889 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:41707
I20260809 16:39:46.255052 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:42681 (local address 127.28.62.188:42983)
0809 16:39:46.230931 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.230934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.231151 (+ 217us) server_negotiation.cc:407] Connection header received
0809 16:39:46.232030 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.232031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.232088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.232143 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.233075 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.233839 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.247271 (+ 13432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.247639 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.247687 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.250673 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.250675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.250676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.250684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.252838 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.254096 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.254097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.254097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.254165 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.254411 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.254412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.254412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.254554 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.254598 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.254837 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.254926 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:46.255074 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.230597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.28.62.130:42681)
0809 16:39:46.231093 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.231103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.231935 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.232229 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.232232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.232780 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.232935 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.232941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.233964 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.233964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.247151 (+ 13187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.247154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.247775 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.247775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.247849 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.248584 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.248589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.250558 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.253686 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.253687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.253688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.253976 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.254270 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.254270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.254271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.254333 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.254668 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.254669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.254711 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.254810 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.254918 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":93,"thread_start_us":77,"threads_started":1}
I20260809 16:39:46.255559 4805 heartbeater.cc:347] Connected to a master server at 127.28.62.188:42983
I20260809 16:39:46.255625 4805 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.255699 4805 heartbeater.cc:511] Master 127.28.62.188:42983 requested a full tablet report, sending...
I20260809 16:39:46.255959 4445 ts_manager.cc:194] Registered new tserver with Master: 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
W20260809 16:39:46.336592 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:46.336846 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:46.336889 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:46.338047 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:46.338124 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:46.338167 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:46.339659 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:36003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=39325
--tserver_master_addrs=127.28.62.190:35925,127.28.62.189:38341,127.28.62.188:42983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:46.339926 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:46.340157 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:46.345963 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:46.346035 4812 server_base.cc:1034] running on GCE node
W20260809 16:39:46.345930 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:46.345990 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:46.346442 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:46.346680 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:46.347854 4812 hybrid_clock.cc:648] HybridClock initialized: now 1786293586347836 us; error 31 us; skew 500 ppm
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:46.350493 4812 init.cc:377] Logged in from keytab as oryx/127.28.62.131@KRBTEST.COM (short username oryx)
I20260809 16:39:46.350907 4812 webserver.cc:460] Webserver started at http://127.28.62.131:39325/ using document root <none> and password file <none>
I20260809 16:39:46.351181 4812 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:46.351244 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:46.352685 4812 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:46.353297 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.353506 4812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:46.353596 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "a03d602c44f540a5a04aabcc2a423736"
format_stamp: "Formatted at 2026-08-09 16:39:45 on dist-test-slave-wl2h"
server_key: "6dcf3bef600ab1d99f232e2714a802cc"
server_key_iv: "fb48b8c08f57ae051c983359719556f4"
server_key_version: "encryptionkey@0"
I20260809 16:39:46.353989 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:46.363346 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.360236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:42980)
0809 16:39:46.360337 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.360350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.360422 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.360595 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.360595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.360597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.360643 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.360645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.361513 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.361514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.362324 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.362326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.363131 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.363133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.363186 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.363189 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.363239 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.363284 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 16:39:46.363365 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.360314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.28.62.188:42983)
0809 16:39:46.360357 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.360359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.360411 (+ 52us) server_negotiation.cc:407] Connection header received
0809 16:39:46.360475 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.360476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.360501 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.360588 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.360707 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.361432 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.362409 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.362993 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.363091 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.363155 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.363260 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.363297 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:46.363838 4455 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 16:39:46.364009 4487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8887286f7849429b5e0e84d1b4184a. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:46.364171 4487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: LEARNER
I20260809 16:39:46.366545 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:46.368831 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:46.369001 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:46.369266 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:46.369598 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:46.369656 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.369700 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:46.369724 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:46.376022 4812 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36003
I20260809 16:39:46.376120 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36003 every 8 connection(s)
I20260809 16:39:46.376437 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:46.378741 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 4812
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.188@KRBTEST.COM
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.189@KRBTEST.COM
Aug 09 16:39:46 dist-test-slave-wl2h krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293586, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:46.389194 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:58917 (local address 127.28.62.188:42983)
0809 16:39:46.377351 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.377353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.377779 (+ 426us) server_negotiation.cc:407] Connection header received
0809 16:39:46.378272 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.378274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.378286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.378344 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.379463 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380037 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381049 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381254 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381329 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.385868 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.385869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.385869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.385876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.387629 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388131 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.388132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.388133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.388193 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388437 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.388438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.388438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.388576 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.388619 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.389015 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.389096 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:39:46.389509 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.28.62.131:58917)
0809 16:39:46.377491 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.377723 (+ 232us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.378175 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.378820 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.378821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.379079 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.379330 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.379338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380157 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.380158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.380942 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.380944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381350 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.381480 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.382262 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.382272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.385771 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.387725 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.387726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.387726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.388021 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.388275 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.388276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.388361 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.388670 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.388678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.388851 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.388972 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.389288 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":217,"thread_start_us":97,"threads_started":1}
I20260809 16:39:46.389814 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:50321 (local address 127.28.62.189:38341)
0809 16:39:46.377414 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.377416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.377646 (+ 230us) server_negotiation.cc:407] Connection header received
0809 16:39:46.378314 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.378315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.378330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.378365 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.379456 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380311 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381560 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381825 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381922 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.385147 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.385149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.385149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.385157 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.387372 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388057 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.388059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.388059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.388122 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388436 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.388437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.388438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.388593 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.388662 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.389588 (+ 926us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.389684 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 16:39:46.390136 4960 heartbeater.cc:347] Connected to a master server at 127.28.62.188:42983
I20260809 16:39:46.390233 4960 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.390617 4960 heartbeater.cc:511] Master 127.28.62.188:42983 requested a full tablet report, sending...
I20260809 16:39:46.390363 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:36709)
0809 16:39:46.377633 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.377642 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.378165 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.378455 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.378458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.379030 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.379329 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.379335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380423 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.380424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.381470 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.381473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381972 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.382039 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.382582 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.382586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.385441 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.388561 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.388562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.388783 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.389295 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.389296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.389296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.389371 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.389971 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.389972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.390026 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.390113 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.390227 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":269,"thread_start_us":67,"threads_started":1}
I20260809 16:39:46.391136 4445 ts_manager.cc:194] Registered new tserver with Master: a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:46.391209 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:36709 (local address 127.28.62.190:35925)
0809 16:39:46.377372 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.377374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.377719 (+ 345us) server_negotiation.cc:407] Connection header received
0809 16:39:46.378303 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.378305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.378320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.378359 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.379456 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380310 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381579 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381899 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381979 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.385519 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.385520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.385521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.385529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.388474 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.389132 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.389133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.389133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.389205 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.389452 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.389453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.389453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.389588 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.389655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.390995 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.391105 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260809 16:39:46.391500 4961 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35925
I20260809 16:39:46.391556 4961 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.391626 4961 heartbeater.cc:511] Master 127.28.62.190:35925 requested a full tablet report, sending...
I20260809 16:39:46.391842 3530 ts_manager.cc:194] Re-registered known tserver with Master: a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:46.392108 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.382530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.28.62.188:42983)
0809 16:39:46.383560 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.383562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.383919 (+ 357us) server_negotiation.cc:407] Connection header received
0809 16:39:46.383973 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.383974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.383988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.384056 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.384690 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.385440 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.386547 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.386867 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.386930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.388559 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.388559 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.388560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.388566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.390702 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.391128 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.391129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.391130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.391225 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.391434 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.391435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.391436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.391557 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.391659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.391862 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.391992 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260809 16:39:46.392385 3530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:36709
I20260809 16:39:46.392992 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.377200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.28.62.131:50321)
0809 16:39:46.377509 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.377540 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.378165 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.378452 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.378457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.378955 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:46.379329 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.379335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.380423 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.380424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.381477 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.381479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.381970 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.381972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.382046 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.382695 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:46.382699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:46.385042 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:46.387493 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.387500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.387503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.387964 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.388203 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:46.388204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:46.388205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:46.388352 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:46.388690 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:46.388691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:46.388852 (+ 161us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.388967 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.389206 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260809 16:39:46.393404 4963 heartbeater.cc:347] Connected to a master server at 127.28.62.189:38341
I20260809 16:39:46.393456 4963 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:46.393532 4963 heartbeater.cc:511] Master 127.28.62.189:38341 requested a full tablet report, sending...
I20260809 16:39:46.393810 4233 ts_manager.cc:194] Registered new tserver with Master: a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:46.398833 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.382837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.28.62.190:35925)
0809 16:39:46.383246 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.383247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.383254 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:46.383278 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.383279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.383315 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.383391 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.390545 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.391923 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.393660 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.393932 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.394065 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.395709 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.395710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.395711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.395718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.397407 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.397855 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.397856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.397857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.397928 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.398173 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.398174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.398174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.398316 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.398431 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.398606 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.398728 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":86,"threads_started":1}
I20260809 16:39:46.399430 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.383118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.28.62.189:38341)
0809 16:39:46.383468 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.383470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.383477 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:46.383496 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.383497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.383517 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.383594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.384405 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.385238 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.393993 (+ 8755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.394254 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.394364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.396092 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.396093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.396093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.396101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.398002 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.398435 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.398436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.398436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.398505 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.398761 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.398763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.398763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.398899 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.398999 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.399179 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.399295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":57,"threads_started":1}
I20260809 16:39:46.400611 3530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:46.402791 3530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:46.403690 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: NON_VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: true } } }
I20260809 16:39:46.403791 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:46.405318 4983 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: attempting to promote NON_VOTER 3aa246cdeb6841d186c1f965a627f5d7 to VOTER
I20260809 16:39:46.405462 4983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } }
I20260809 16:39:46.405864 4245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Refusing update from remote peer 3e8887286f7849429b5e0e84d1b4184a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 16:39:46.406038 4183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 16:39:46.406641 4983 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.406754 4455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 LEARNER]: Refusing update from remote peer 3e8887286f7849429b5e0e84d1b4184a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 16:39:46.406797 4245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.407744 4983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 16:39:46.407810 4404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.407893 4404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:39:46.407876 4983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.407941 4983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:46.408041 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.408108 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:46.408357 4455 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.409260 4487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.409369 4487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:39:46.410465 4183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8887286f7849429b5e0e84d1b4184a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:46.410570 4183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:46.414569 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.410192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.28.62.131:36003)
0809 16:39:46.410351 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.410355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.410365 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:46.410429 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.410431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.410449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.410697 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.410764 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.411903 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.413503 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414188 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.414280 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.414363 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.414438 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.414484 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260809 16:39:46.414572 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.409779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:37471 (local address 127.0.0.1:41510)
0809 16:39:46.410104 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.410118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.410173 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.410966 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.410967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.410970 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.411007 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.411032 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.412231 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.412232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.413349 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.413351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414349 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.414350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.414422 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.414426 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.414459 (+ 33us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.414495 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260809 16:39:46.414572 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.409883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36003 (local address 127.0.0.1:53586)
0809 16:39:46.410116 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.410131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.410178 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.410559 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.410560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.410562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.410614 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.410616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.412001 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.412003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.413412 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.413416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414278 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.414279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.414347 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.414352 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.414424 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.414467 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260809 16:39:46.415154 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.409946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40813 (local address 127.0.0.1:40664)
0809 16:39:46.410506 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.410518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.410567 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.411043 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.411044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.411047 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.411099 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.411100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.411996 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.411997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.413467 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.413469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414857 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.414858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.414907 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.414910 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.414999 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.415071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":51,"thread_start_us":108,"threads_started":1}
I20260809 16:39:46.415257 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.410071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.28.62.130:40813)
0809 16:39:46.410862 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.410866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.410876 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:46.410909 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.410910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.410925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.411028 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.411163 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.411894 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.413565 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414339 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.414416 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.414820 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.415105 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.415154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":119,"threads_started":1}
I20260809 16:39:46.415730 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.410283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.28.62.129:37471)
0809 16:39:46.410797 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.410803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.410812 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:46.410849 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.410851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.410866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.411108 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.411170 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.412166 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.413441 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.414262 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.415063 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.415166 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.415189 (+ 23us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.415530 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":119,"threads_started":1}
I20260809 16:39:46.415699 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 1063f731db02421ba00b7e5a7dedd189 (DEFAULT_TABLE table=test-table [id=123d028ded584cdb918492675fc6988c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:46.415935 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 1063f731db02421ba00b7e5a7dedd189 (DEFAULT_TABLE table=test-table [id=123d028ded584cdb918492675fc6988c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:46.416127 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1063f731db02421ba00b7e5a7dedd189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:46.416294 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1063f731db02421ba00b7e5a7dedd189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:46.420187 4997 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Bootstrap starting.
I20260809 16:39:46.420847 4997 tablet_bootstrap.cc:654] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:46.420830 4575 tablet_service.cc:1467] Processing CreateTablet for tablet 1063f731db02421ba00b7e5a7dedd189 (DEFAULT_TABLE table=test-table [id=123d028ded584cdb918492675fc6988c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:46.421185 4997 log.cc:824] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:46.421228 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1063f731db02421ba00b7e5a7dedd189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:46.421511 4998 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Bootstrap starting.
I20260809 16:39:46.422055 4998 tablet_bootstrap.cc:654] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:46.422046 4997 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: No bootstrap required, opened a new log
I20260809 16:39:46.422143 4997 ts_tablet_manager.cc:1397] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:46.422381 4998 log.cc:824] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:46.423591 4998 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: No bootstrap required, opened a new log
I20260809 16:39:46.423676 4998 ts_tablet_manager.cc:1397] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:46.424474 4997 raft_consensus.cc:348] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.424679 4997 raft_consensus.cc:374] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:46.424741 4997 raft_consensus.cc:729] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ba9869902c436e9d40f1df25fdebec, State: Initialized, Role: FOLLOWER
I20260809 16:39:46.424887 4997 consensus_queue.cc:260] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.425184 4997 ts_tablet_manager.cc:1428] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:39:46.425361 4806 heartbeater.cc:503] Master 127.28.62.190:35925 was elected leader, sending a full tablet report...
I20260809 16:39:46.425729 5001 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Bootstrap starting.
I20260809 16:39:46.426429 5001 tablet_bootstrap.cc:654] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:46.426817 5001 log.cc:824] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:46.426882 4998 raft_consensus.cc:348] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.427075 4998 raft_consensus.cc:374] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:46.427129 4998 raft_consensus.cc:729] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03d602c44f540a5a04aabcc2a423736, State: Initialized, Role: FOLLOWER
I20260809 16:39:46.427253 4998 consensus_queue.cc:260] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.427517 4998 ts_tablet_manager.cc:1428] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 16:39:46.427588 4961 heartbeater.cc:503] Master 127.28.62.190:35925 was elected leader, sending a full tablet report...
I20260809 16:39:46.428208 5001 tablet_bootstrap.cc:492] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: No bootstrap required, opened a new log
I20260809 16:39:46.428278 5001 ts_tablet_manager.cc:1397] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:46.429801 5001 raft_consensus.cc:348] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.429944 5001 raft_consensus.cc:374] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:46.429991 5001 raft_consensus.cc:729] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c985aba99f8496d84710ebd3f5a3735, State: Initialized, Role: FOLLOWER
I20260809 16:39:46.430099 5001 consensus_queue.cc:260] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.430341 5001 ts_tablet_manager.cc:1428] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:39:46.430436 4648 heartbeater.cc:503] Master 127.28.62.190:35925 was elected leader, sending a full tablet report...
W20260809 16:39:46.480872 4808 tablet.cc:2369] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:39:46.581849 4651 tablet.cc:2369] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:46.618113 5002 raft_consensus.cc:482] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:46.618251 5002 raft_consensus.cc:504] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.618661 5002 leader_election.cc:288] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471), a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:46.621622 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.618848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:37471 (local address 127.28.62.130:53461)
0809 16:39:46.618917 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.618931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.619046 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.619226 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.619227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.619228 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.619341 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.619344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.620005 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.620006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.620801 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.620803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.621330 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.621331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.621414 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.621417 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.621470 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.621502 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":29}
I20260809 16:39:46.621621 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.618983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:53461 (local address 127.28.62.129:37471)
0809 16:39:46.619052 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.619054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.619065 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:39:46.619135 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.619136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.619144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.619200 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.619436 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.619923 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.620902 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.621250 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.621307 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.621314 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.621502 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.621548 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 16:39:46.621675 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.619109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:37459 (local address 127.28.62.131:36003)
0809 16:39:46.619161 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.619163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.619177 (+ 14us) server_negotiation.cc:407] Connection header received
0809 16:39:46.619225 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.619226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.619235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.619295 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:46.619471 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.619928 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.620906 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.621317 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.621387 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.621396 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.621557 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.621600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 16:39:46.621721 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.618962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36003 (local address 127.28.62.130:37459)
0809 16:39:46.619028 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:46.619041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:46.619082 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:46.619308 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:46.619309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:46.619310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:46.619368 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.619370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.620028 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.620029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.620801 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:46.620803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.621397 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.621398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:46.621475 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.621479 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:46.621537 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:39:46.621572 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":24}
I20260809 16:39:46.622052 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1063f731db02421ba00b7e5a7dedd189" candidate_uuid: "57ba9869902c436e9d40f1df25fdebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03d602c44f540a5a04aabcc2a423736" is_pre_election: true
I20260809 16:39:46.622048 4596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1063f731db02421ba00b7e5a7dedd189" candidate_uuid: "57ba9869902c436e9d40f1df25fdebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c985aba99f8496d84710ebd3f5a3735" is_pre_election: true
I20260809 16:39:46.622216 4908 raft_consensus.cc:2454] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ba9869902c436e9d40f1df25fdebec in term 0.
I20260809 16:39:46.622229 4596 raft_consensus.cc:2454] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ba9869902c436e9d40f1df25fdebec in term 0.
I20260809 16:39:46.622459 4680 leader_election.cc:302] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ba9869902c436e9d40f1df25fdebec, 8c985aba99f8496d84710ebd3f5a3735; no voters:
I20260809 16:39:46.622587 5002 raft_consensus.cc:2781] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:46.622642 5002 raft_consensus.cc:482] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:46.622677 5002 raft_consensus.cc:3037] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:46.623442 5002 raft_consensus.cc:504] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.623572 5002 leader_election.cc:288] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [CANDIDATE]: Term 1 election: Requested vote from peers 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471), a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003)
I20260809 16:39:46.623726 4596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1063f731db02421ba00b7e5a7dedd189" candidate_uuid: "57ba9869902c436e9d40f1df25fdebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c985aba99f8496d84710ebd3f5a3735"
I20260809 16:39:46.623785 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1063f731db02421ba00b7e5a7dedd189" candidate_uuid: "57ba9869902c436e9d40f1df25fdebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03d602c44f540a5a04aabcc2a423736"
I20260809 16:39:46.623814 4596 raft_consensus.cc:3037] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:46.623850 4908 raft_consensus.cc:3037] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:46.624706 4908 raft_consensus.cc:2454] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ba9869902c436e9d40f1df25fdebec in term 1.
I20260809 16:39:46.624706 4596 raft_consensus.cc:2454] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ba9869902c436e9d40f1df25fdebec in term 1.
I20260809 16:39:46.624900 4680 leader_election.cc:302] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ba9869902c436e9d40f1df25fdebec, a03d602c44f540a5a04aabcc2a423736; no voters:
I20260809 16:39:46.625046 5002 raft_consensus.cc:2781] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:46.625285 5002 raft_consensus.cc:686] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 1 LEADER]: Becoming Leader. State: Replica: 57ba9869902c436e9d40f1df25fdebec, State: Running, Role: LEADER
I20260809 16:39:46.625536 5002 consensus_queue.cc:237] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:46.626423 3530 catalog_manager.cc:5495] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ba9869902c436e9d40f1df25fdebec (127.28.62.130). New cstate: current_term: 1 leader_uuid: "57ba9869902c436e9d40f1df25fdebec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } health_report { overall_health: HEALTHY } } }
W20260809 16:39:46.627220 4964 tablet.cc:2369] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:46.641757 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.639275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.28.62.130:40813)
0809 16:39:46.639357 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.639360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.639370 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:46.639450 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.639451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.639471 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.639553 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:46.639702 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.640199 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.640966 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.641171 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.641241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.641422 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:46.641529 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:46.641619 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.641646 (+ 27us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.641669 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 16:39:46.643920 4908 raft_consensus.cc:1261] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Refusing update from remote peer 57ba9869902c436e9d40f1df25fdebec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:46.643913 4596 raft_consensus.cc:1261] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Refusing update from remote peer 57ba9869902c436e9d40f1df25fdebec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:46.644351 5006 consensus_queue.cc:1035] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:46.644505 5002 consensus_queue.cc:1035] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:46.647274 5009 mvcc.cc:205] Tried to move back new op lower bound from 7316658530890330112 to 7316658530818859008. Current Snapshot: MvccSnapshot[applied={T|T < 7316658530890330112}]
I20260809 16:39:46.648706 5010 mvcc.cc:205] Tried to move back new op lower bound from 7316658530890330112 to 7316658530818859008. Current Snapshot: MvccSnapshot[applied={T|T < 7316658530890330112}]
I20260809 16:39:46.649111 5011 mvcc.cc:205] Tried to move back new op lower bound from 7316658530890330112 to 7316658530818859008. Current Snapshot: MvccSnapshot[applied={T|T < 7316658530890330112}]
I20260809 16:39:46.650519 3530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:46.650622 3530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:46.651963 3530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e8887286f7849429b5e0e84d1b4184a: Sending DeleteTablet for 3 replicas of tablet 1063f731db02421ba00b7e5a7dedd189
I20260809 16:39:46.652316 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1063f731db02421ba00b7e5a7dedd189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:46 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:53586
I20260809 16:39:46.652349 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 1063f731db02421ba00b7e5a7dedd189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:46 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:40664
I20260809 16:39:46.652570 5022 tablet_replica.cc:330] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: stopping tablet replica
I20260809 16:39:46.652561 4575 tablet_service.cc:1514] Processing DeleteTablet for tablet 1063f731db02421ba00b7e5a7dedd189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:46 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:41510
I20260809 16:39:46.652695 5022 raft_consensus.cc:2229] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:46.652814 5022 raft_consensus.cc:2258] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:46.653720 5022 ts_tablet_manager.cc:1905] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:46.653733 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 3491
I20260809 16:39:46.655567 5022 ts_tablet_manager.cc:1918] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:46.655632 5022 log.cc:1196] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/1063f731db02421ba00b7e5a7dedd189
I20260809 16:39:46.655848 5022 ts_tablet_manager.cc:1939] T 1063f731db02421ba00b7e5a7dedd189 P 57ba9869902c436e9d40f1df25fdebec: Deleting consensus metadata
I20260809 16:39:46.664013 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.653291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.28.62.188:42983)
0809 16:39:46.653342 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.653344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.653456 (+ 112us) server_negotiation.cc:407] Connection header received
0809 16:39:46.653555 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.653556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.653570 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.653600 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.658318 (+ 4718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.658855 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.659652 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.659860 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.659938 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.661238 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.661240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.661240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.661247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.662708 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.663174 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.663176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.663176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.663239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.663451 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.663452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.663452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.663569 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.663680 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.663784 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.663870 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:46.667198 5024 tablet_replica.cc:330] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: stopping tablet replica
W20260809 16:39:46.667192 4677 connection.cc:535] server connection from 127.0.0.1:40664 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 16:39:46.667382 5020 tablet_replica.cc:330] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: stopping tablet replica
I20260809 16:39:46.667534 5020 raft_consensus.cc:2229] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:46.667672 5020 raft_consensus.cc:2258] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:46.667856 5024 raft_consensus.cc:2229] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:46.667980 5024 raft_consensus.cc:2258] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:46.668535 5024 ts_tablet_manager.cc:1905] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:46.668535 5020 ts_tablet_manager.cc:1905] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:46.670516 5020 ts_tablet_manager.cc:1918] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:46.670610 5024 ts_tablet_manager.cc:1918] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:46.670629 5020 log.cc:1196] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/1063f731db02421ba00b7e5a7dedd189
I20260809 16:39:46.670694 5024 log.cc:1196] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/1063f731db02421ba00b7e5a7dedd189
I20260809 16:39:46.670960 5020 ts_tablet_manager.cc:1939] T 1063f731db02421ba00b7e5a7dedd189 P a03d602c44f540a5a04aabcc2a423736: Deleting consensus metadata
I20260809 16:39:46.671075 5024 ts_tablet_manager.cc:1939] T 1063f731db02421ba00b7e5a7dedd189 P 8c985aba99f8496d84710ebd3f5a3735: Deleting consensus metadata
W20260809 16:39:46.671531 4519 connection.cc:441] server connection from 127.0.0.1:41510 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41510 (request call id 1) could send its response
W20260809 16:39:46.671561 4835 connection.cc:441] server connection from 127.0.0.1:53586 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53586 (request call id 1) could send its response
I20260809 16:39:46.678136 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.670492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.28.62.189:38341)
0809 16:39:46.670557 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.670560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.671448 (+ 888us) server_negotiation.cc:407] Connection header received
0809 16:39:46.671573 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.671574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.671591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.671624 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.672334 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.672842 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.673557 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.673776 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.673852 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.675294 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.675295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.675295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.675302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.676828 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.677187 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.677188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.677189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.677252 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.677473 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.677474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.677474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.677616 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.677727 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.677859 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.677935 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:39:46.679785 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:46.670841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.28.62.188:42983)
0809 16:39:46.670884 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:46.670886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:46.671763 (+ 877us) server_negotiation.cc:407] Connection header received
0809 16:39:46.671865 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:46.671866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:46.671877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:46.671907 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:46.673032 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.673850 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.674907 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:46.675230 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:46.675272 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:46.676813 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:46.676815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:46.676815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:46.676823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:46.678591 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.678907 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.678908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.678908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.678971 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:46.679249 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:46.679250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:46.679250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:46.679384 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:46.679484 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:46.679589 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 16:39:46.679672 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:47.648782 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:47.648419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:37851)
0809 16:39:47.648636 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:47.648719 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260809 16:39:47.648913 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:47.648457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:34285)
0809 16:39:47.648760 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:47.648857 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":63,"threads_started":1}
I20260809 16:39:47.648983 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:47.648735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:36689)
0809 16:39:47.648865 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:47.648934 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":52,"threads_started":1}
W20260809 16:39:47.649168 4648 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
W20260809 16:39:47.649227 4806 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
W20260809 16:39:47.649731 4961 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.152349 5037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e8887286f7849429b5e0e84d1b4184a)
I20260809 16:39:48.152499 5037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } }
I20260809 16:39:48.152868 5037 leader_election.cc:288] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925), 3aa246cdeb6841d186c1f965a627f5d7 (127.28.62.188:42983)
I20260809 16:39:48.153390 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43108)
0809 16:39:48.153258 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.153344 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260809 16:39:48.153422 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e8887286f7849429b5e0e84d1b4184a)
W20260809 16:39:48.153522 4216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.153513 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } }
I20260809 16:39:48.153926 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43122)
0809 16:39:48.153866 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.153893 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":88,"threads_started":1}
W20260809 16:39:48.154016 4216 leader_election.cc:334] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925): Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.154161 5041 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925), 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189:38341)
I20260809 16:39:48.154371 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43132)
0809 16:39:48.154267 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.154333 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":49,"threads_started":1}
W20260809 16:39:48.154506 4431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.154891 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.154814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43144)
0809 16:39:48.154845 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.154868 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260809 16:39:48.155054 4431 leader_election.cc:334] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925): Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.157081 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.154039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.28.62.189:38341)
0809 16:39:48.154200 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.154202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.154211 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:48.154280 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.154280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.154290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.154377 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.154522 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.155267 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.156221 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.156756 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.156814 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.156823 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.156999 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.157033 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260809 16:39:48.157223 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:38341 (local address 127.0.0.1:47388)
0809 16:39:48.154152 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.154164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.154225 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.154398 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.154399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.154402 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.154452 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.154454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.155361 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.155362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.156133 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.156135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.156837 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.156838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.156927 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.156931 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.156981 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.157023 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":44,"threads_started":1}
I20260809 16:39:48.157536 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa246cdeb6841d186c1f965a627f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87a04e4b3a9b44faab8a5025613e2291" is_pre_election: true
I20260809 16:39:48.157665 4245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa246cdeb6841d186c1f965a627f5d7 in term 1.
I20260809 16:39:48.157835 4429 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aa246cdeb6841d186c1f965a627f5d7, 87a04e4b3a9b44faab8a5025613e2291; no voters: 3e8887286f7849429b5e0e84d1b4184a
I20260809 16:39:48.158021 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 16:39:48.158089 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e8887286f7849429b5e0e84d1b4184a)
I20260809 16:39:48.158191 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:39:48.158516 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.28.62.188:42983)
0809 16:39:48.153555 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.153558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.153754 (+ 196us) server_negotiation.cc:407] Connection header received
0809 16:39:48.153843 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.153845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.153858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.153929 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.154373 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.155269 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.157653 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.158170 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.158246 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.158259 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.158436 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.158474 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260809 16:39:48.158542 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.153159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:42983 (local address 127.0.0.1:43020)
0809 16:39:48.153684 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.153696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.153779 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.154127 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.154128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.154131 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.154198 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.154200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.156222 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.156223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.157558 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.157562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.158296 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.158297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.158364 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.158369 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.158427 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.158469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":30,"thread_start_us":41,"threads_started":1}
I20260809 16:39:48.158824 4455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87a04e4b3a9b44faab8a5025613e2291" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aa246cdeb6841d186c1f965a627f5d7" is_pre_election: true
I20260809 16:39:48.159221 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } }
I20260809 16:39:48.159345 4455 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87a04e4b3a9b44faab8a5025613e2291 in current term 2: Already voted for candidate 3aa246cdeb6841d186c1f965a627f5d7 in this term.
I20260809 16:39:48.159369 5041 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 election: Requested vote from peers 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925), 87a04e4b3a9b44faab8a5025613e2291 (127.28.62.189:38341)
I20260809 16:39:48.159571 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa246cdeb6841d186c1f965a627f5d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87a04e4b3a9b44faab8a5025613e2291"
I20260809 16:39:48.159657 4215 leader_election.cc:302] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87a04e4b3a9b44faab8a5025613e2291; no voters: 3aa246cdeb6841d186c1f965a627f5d7, 3e8887286f7849429b5e0e84d1b4184a
I20260809 16:39:48.159669 4245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:39:48.159858 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.159510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43146)
0809 16:39:48.159764 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.159801 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
I20260809 16:39:48.160199 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.160083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43160)
0809 16:39:48.160149 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.160175 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 16:39:48.160310 4431 leader_election.cc:334] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925): Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
I20260809 16:39:48.160585 4245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa246cdeb6841d186c1f965a627f5d7 in term 2.
I20260809 16:39:48.160686 5037 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:39:48.160772 4429 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aa246cdeb6841d186c1f965a627f5d7, 87a04e4b3a9b44faab8a5025613e2291; no voters: 3e8887286f7849429b5e0e84d1b4184a
I20260809 16:39:48.160909 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 16:39:48.161037 5041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [term 2 LEADER]: Becoming Leader. State: Replica: 3aa246cdeb6841d186c1f965a627f5d7, State: Running, Role: LEADER
I20260809 16:39:48.161145 5041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } }
I20260809 16:39:48.161715 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa246cdeb6841d186c1f965a627f5d7. Latest consensus state: current_term: 2 leader_uuid: "3aa246cdeb6841d186c1f965a627f5d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:48.161813 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:48.162029 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:48.162432 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:48.162490 5053 catalog_manager.cc:1259] Loaded cluster ID: af1362a1d1a04665a414211f86a803cf
I20260809 16:39:48.162544 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:48.162716 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:48.162815 5053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Loaded TSK: 0
I20260809 16:39:48.162959 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:39:48.222409 4444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:48.224334 4444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:48.225595 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.225453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43174)
0809 16:39:48.225519 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.225566 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 16:39:48.225617 4245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [term 2 FOLLOWER]: Refusing update from remote peer 3aa246cdeb6841d186c1f965a627f5d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 16:39:48.225955 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.225877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.0.0.1:43182)
0809 16:39:48.225906 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.225935 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260809 16:39:48.225894 5051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 16:39:48.226049 4431 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 -> Peer 3e8887286f7849429b5e0e84d1b4184a (127.28.62.190:35925): Couldn't send request to peer 3e8887286f7849429b5e0e84d1b4184a. Status: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 16:39:48.226521 5037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa246cdeb6841d186c1f965a627f5d7. Latest consensus state: current_term: 2 leader_uuid: "3aa246cdeb6841d186c1f965a627f5d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:48.226620 5037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:39:48.227015 5054 mvcc.cc:205] Tried to move back new op lower bound from 7316658537369493504 to 7316658537108881408. Current Snapshot: MvccSnapshot[applied={T|T < 7316658537369493504}]
I20260809 16:39:48.227219 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3aa246cdeb6841d186c1f965a627f5d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:48.227330 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:48.227483 5037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3aa246cdeb6841d186c1f965a627f5d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e8887286f7849429b5e0e84d1b4184a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35925 } } peers { permanent_uuid: "87a04e4b3a9b44faab8a5025613e2291" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 38341 } attrs { promote: false } } peers { permanent_uuid: "3aa246cdeb6841d186c1f965a627f5d7" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 42983 } attrs { promote: false } } }
I20260809 16:39:48.227578 5037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87a04e4b3a9b44faab8a5025613e2291 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:39:48.232411 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36003 (local address 127.0.0.1:53592)
0809 16:39:48.229342 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.229355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.229415 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.229618 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.229619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.229622 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.229690 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.229693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.230569 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.230570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.231425 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.231426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.232187 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.232188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.232254 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.232258 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.232307 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.232345 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":3}
I20260809 16:39:48.232625 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.28.62.131:36003)
0809 16:39:48.229468 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.229471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.229479 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:48.229508 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.229510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.229521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.229759 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.229799 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.230508 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.231503 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.232107 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.232187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.232201 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.232515 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.232558 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260809 16:39:48.232842 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 318813bfb6a24d23998eaad9bad451d4 (DEFAULT_TABLE table=test-table [id=bfa8d40535f643fc95da6823171b2362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:48.232990 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318813bfb6a24d23998eaad9bad451d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:48.233273 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40813 (local address 127.0.0.1:40690)
0809 16:39:48.229456 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.229467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.229508 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.230444 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.230445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.230447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.230496 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.230497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.231409 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.231410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.232196 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.232199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.233011 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.233012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.233080 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.233085 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.233135 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.233181 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260809 16:39:48.233369 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.28.62.129:37471)
0809 16:39:48.229757 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.229759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.229768 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:48.229795 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.229796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.229807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.229874 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.229979 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.230728 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.232038 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.233024 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.233092 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.233105 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.233244 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.233289 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260809 16:39:48.233372 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:37471 (local address 127.0.0.1:41518)
0809 16:39:48.229647 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.229657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.229696 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.229888 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.229889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.229891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.229921 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.229923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.230815 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.230816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.231946 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.231949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.233093 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.233095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.233160 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.233165 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.233226 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.233265 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":95,"threads_started":1}
I20260809 16:39:48.233676 4575 tablet_service.cc:1467] Processing CreateTablet for tablet 318813bfb6a24d23998eaad9bad451d4 (DEFAULT_TABLE table=test-table [id=bfa8d40535f643fc95da6823171b2362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:48.233810 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318813bfb6a24d23998eaad9bad451d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:48.233916 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.229500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.28.62.130:40813)
0809 16:39:48.230297 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.230299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.230308 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:48.230331 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.230332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.230342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.230565 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.230600 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.231324 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.232267 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.232925 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.232991 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.233003 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.233797 (+ 794us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.233844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":66,"threads_started":1}
I20260809 16:39:48.234534 5066 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Bootstrap starting.
I20260809 16:39:48.234658 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 318813bfb6a24d23998eaad9bad451d4 (DEFAULT_TABLE table=test-table [id=bfa8d40535f643fc95da6823171b2362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:48.234786 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318813bfb6a24d23998eaad9bad451d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:48.235162 5066 tablet_bootstrap.cc:654] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:48.235351 5067 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735: Bootstrap starting.
I20260809 16:39:48.235667 5067 tablet_bootstrap.cc:654] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:48.235996 5066 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: No bootstrap required, opened a new log
I20260809 16:39:48.236061 5066 ts_tablet_manager.cc:1397] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 16:39:48.236210 5070 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Bootstrap starting.
I20260809 16:39:48.236207 5066 raft_consensus.cc:348] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.236282 5066 raft_consensus.cc:374] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:48.236304 5066 raft_consensus.cc:729] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03d602c44f540a5a04aabcc2a423736, State: Initialized, Role: FOLLOWER
I20260809 16:39:48.236366 5066 consensus_queue.cc:260] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.236577 5070 tablet_bootstrap.cc:654] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:48.236582 5066 ts_tablet_manager.cc:1428] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:48.236968 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.236672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:39633)
0809 16:39:48.236891 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.236933 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260809 16:39:48.236995 5067 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735: No bootstrap required, opened a new log
I20260809 16:39:48.237164 5067 ts_tablet_manager.cc:1397] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:48.237231 4960 heartbeater.cc:503] Master 127.28.62.188:42983 was elected leader, sending a full tablet report...
I20260809 16:39:48.237361 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.237194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:33981)
0809 16:39:48.237311 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.237334 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260809 16:39:48.237720 5067 raft_consensus.cc:348] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.237807 5067 raft_consensus.cc:374] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:48.237823 5067 raft_consensus.cc:729] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c985aba99f8496d84710ebd3f5a3735, State: Initialized, Role: FOLLOWER
I20260809 16:39:48.237913 5070 tablet_bootstrap.cc:492] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: No bootstrap required, opened a new log
I20260809 16:39:48.237902 5067 consensus_queue.cc:260] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.237967 5070 ts_tablet_manager.cc:1397] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:48.238065 5067 ts_tablet_manager.cc:1428] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:48.238113 5070 raft_consensus.cc:348] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.238194 5070 raft_consensus.cc:374] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:48.238217 5070 raft_consensus.cc:729] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ba9869902c436e9d40f1df25fdebec, State: Initialized, Role: FOLLOWER
I20260809 16:39:48.238279 5070 consensus_queue.cc:260] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.238415 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.238208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:34653)
0809 16:39:48.238335 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.238367 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":90,"threads_started":1}
I20260809 16:39:48.238428 5070 ts_tablet_manager.cc:1428] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:48.238842 4647 heartbeater.cc:503] Master 127.28.62.188:42983 was elected leader, sending a full tablet report...
I20260809 16:39:48.238909 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.238617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:44195)
0809 16:39:48.238730 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.238827 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260809 16:39:48.238938 4805 heartbeater.cc:503] Master 127.28.62.188:42983 was elected leader, sending a full tablet report...
I20260809 16:39:48.239324 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.238620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:45831)
0809 16:39:48.239241 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.239288 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":82,"threads_started":1}
I20260809 16:39:48.239754 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.239672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:37009)
0809 16:39:48.239713 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.239737 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260809 16:39:48.477484 5078 raft_consensus.cc:482] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:48.477617 5078 raft_consensus.cc:504] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.478026 5078 leader_election.cc:288] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003), 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
I20260809 16:39:48.482087 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.478209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36003 (local address 127.28.62.129:50861)
0809 16:39:48.478376 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.478385 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.478466 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.478630 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.478632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.478634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.478725 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.478727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.479686 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.479687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.480879 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.480882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.481762 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.481763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.481863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.481867 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.481920 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.481971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260809 16:39:48.482335 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.478278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:50861 (local address 127.28.62.131:36003)
0809 16:39:48.478327 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.478329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.478449 (+ 120us) server_negotiation.cc:407] Connection header received
0809 16:39:48.478523 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.478524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.478536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.478599 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.478835 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.479583 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.480991 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.481628 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.481709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.481723 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.482194 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.482241 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:48.485466 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318813bfb6a24d23998eaad9bad451d4" candidate_uuid: "8c985aba99f8496d84710ebd3f5a3735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03d602c44f540a5a04aabcc2a423736" is_pre_election: true
I20260809 16:39:48.485585 4908 raft_consensus.cc:2454] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c985aba99f8496d84710ebd3f5a3735 in term 0.
I20260809 16:39:48.485805 4523 leader_election.cc:302] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c985aba99f8496d84710ebd3f5a3735, a03d602c44f540a5a04aabcc2a423736; no voters:
I20260809 16:39:48.485954 5078 raft_consensus.cc:2781] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:48.485989 5078 raft_consensus.cc:482] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:48.486016 5078 raft_consensus.cc:3037] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:48.486948 5078 raft_consensus.cc:504] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.487140 5078 leader_election.cc:288] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [CANDIDATE]: Term 1 election: Requested vote from peers a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003), 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813)
I20260809 16:39:48.487351 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318813bfb6a24d23998eaad9bad451d4" candidate_uuid: "8c985aba99f8496d84710ebd3f5a3735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03d602c44f540a5a04aabcc2a423736"
I20260809 16:39:48.487427 4908 raft_consensus.cc:3037] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:48.488219 4908 raft_consensus.cc:2454] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c985aba99f8496d84710ebd3f5a3735 in term 1.
I20260809 16:39:48.488469 4523 leader_election.cc:302] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c985aba99f8496d84710ebd3f5a3735, a03d602c44f540a5a04aabcc2a423736; no voters:
I20260809 16:39:48.488556 5078 raft_consensus.cc:2781] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:48.488735 5078 raft_consensus.cc:686] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [term 1 LEADER]: Becoming Leader. State: Replica: 8c985aba99f8496d84710ebd3f5a3735, State: Running, Role: LEADER
I20260809 16:39:48.488806 5078 consensus_queue.cc:237] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } }
I20260809 16:39:48.488981 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.478314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59625 (local address 127.28.62.130:40813)
0809 16:39:48.478383 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.478386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.478601 (+ 215us) server_negotiation.cc:407] Connection header received
0809 16:39:48.478685 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.478686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.478697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.478767 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:48.478945 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.479713 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.487545 (+ 7832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.488176 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.488235 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.488247 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.488834 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.488894 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:39:48.489327 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.489232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:53753)
0809 16:39:48.489274 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.489301 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 16:39:48.489567 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.478217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:40813 (local address 127.28.62.129:59625)
0809 16:39:48.478546 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.478555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:48.478626 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:48.478776 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:48.478777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:48.478779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:48.478864 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.478865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.485708 (+ 6843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.485710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.487364 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:48.487367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.488475 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.488476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:48.488663 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.488669 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:39:48.489333 (+ 664us) client_negotiation.cc:240] Negotiation successful
0809 16:39:48.489457 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":103}
I20260809 16:39:48.489914 4444 catalog_manager.cc:5495] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129). New cstate: current_term: 1 leader_uuid: "8c985aba99f8496d84710ebd3f5a3735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c985aba99f8496d84710ebd3f5a3735" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:48.490787 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318813bfb6a24d23998eaad9bad451d4" candidate_uuid: "8c985aba99f8496d84710ebd3f5a3735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ba9869902c436e9d40f1df25fdebec"
I20260809 16:39:48.490777 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318813bfb6a24d23998eaad9bad451d4" candidate_uuid: "8c985aba99f8496d84710ebd3f5a3735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ba9869902c436e9d40f1df25fdebec" is_pre_election: true
I20260809 16:39:48.490938 4752 raft_consensus.cc:2454] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c985aba99f8496d84710ebd3f5a3735 in term 0.
I20260809 16:39:48.517275 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.513815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.28.62.129:37471)
0809 16:39:48.513887 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:48.513890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:48.513899 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:48.513949 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:48.513950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:48.513977 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:48.514075 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:48.514177 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.514994 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.516222 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:48.516557 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:48.516648 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:48.516840 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:48.517014 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:48.517093 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:48.517150 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 16:39:48.517177 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:39:48.518605 4751 raft_consensus.cc:3037] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:48.518639 4908 raft_consensus.cc:1261] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Refusing update from remote peer 8c985aba99f8496d84710ebd3f5a3735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:48.518895 5078 consensus_queue.cc:1035] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03d602c44f540a5a04aabcc2a423736" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:48.519731 4751 raft_consensus.cc:1261] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Refusing update from remote peer 8c985aba99f8496d84710ebd3f5a3735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:48.520145 5078 consensus_queue.cc:1035] T 318813bfb6a24d23998eaad9bad451d4 P 8c985aba99f8496d84710ebd3f5a3735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ba9869902c436e9d40f1df25fdebec" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:48.521015 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.520896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:46291)
0809 16:39:48.520942 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.520981 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 16:39:48.521552 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.520928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:37375)
0809 16:39:48.521486 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.521521 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544}
I20260809 16:39:48.522221 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.519560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:41377)
0809 16:39:48.522140 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.522185 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2554}
I20260809 16:39:48.522575 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.522464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.131:33703)
0809 16:39:48.522520 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.522552 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 16:39:48.523428 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.523200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.129:35377)
0809 16:39:48.523366 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.523394 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260809 16:39:48.523847 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:48.523749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35925 (local address 127.28.62.130:44419)
0809 16:39:48.523792 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.523818 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 16:39:48.525401 4445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:48.525498 4445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:48.526975 4445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aa246cdeb6841d186c1f965a627f5d7: Sending DeleteTablet for 3 replicas of tablet 318813bfb6a24d23998eaad9bad451d4
I20260809 16:39:48.527293 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 318813bfb6a24d23998eaad9bad451d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:48 UTC) from {username='oryx', principal='oryx/127.28.62.188@KRBTEST.COM'} at 127.0.0.1:40690
I20260809 16:39:48.527504 5097 tablet_replica.cc:330] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: stopping tablet replica
I20260809 16:39:48.527585 5097 raft_consensus.cc:2229] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:48.527639 5097 raft_consensus.cc:2258] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:48.527633 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 4402
I20260809 16:39:48.529474 5097 ts_tablet_manager.cc:1905] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:48.531669 5097 ts_tablet_manager.cc:1918] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:48.531767 5097 log.cc:1196] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/318813bfb6a24d23998eaad9bad451d4
I20260809 16:39:48.532078 5097 ts_tablet_manager.cc:1939] T 318813bfb6a24d23998eaad9bad451d4 P 57ba9869902c436e9d40f1df25fdebec: Deleting consensus metadata
I20260809 16:39:48.532389 4429 catalog_manager.cc:4841] TS 57ba9869902c436e9d40f1df25fdebec (127.28.62.130:40813): tablet 318813bfb6a24d23998eaad9bad451d4 (table test-table [id=bfa8d40535f643fc95da6823171b2362]) successfully deleted
I20260809 16:39:48.532876 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet 318813bfb6a24d23998eaad9bad451d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:48 UTC) from {username='oryx', principal='oryx/127.28.62.188@KRBTEST.COM'} at 127.0.0.1:53592
I20260809 16:39:48.533310 5098 tablet_replica.cc:330] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: stopping tablet replica
I20260809 16:39:48.533393 5098 raft_consensus.cc:2229] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:48.533452 5098 raft_consensus.cc:2258] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:48.533753 5098 ts_tablet_manager.cc:1905] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:48.534947 5098 ts_tablet_manager.cc:1918] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:48.535064 5098 log.cc:1196] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/318813bfb6a24d23998eaad9bad451d4
I20260809 16:39:48.535305 5098 ts_tablet_manager.cc:1939] T 318813bfb6a24d23998eaad9bad451d4 P a03d602c44f540a5a04aabcc2a423736: Deleting consensus metadata
I20260809 16:39:48.535655 4431 catalog_manager.cc:4841] TS a03d602c44f540a5a04aabcc2a423736 (127.28.62.131:36003): tablet 318813bfb6a24d23998eaad9bad451d4 (table test-table [id=bfa8d40535f643fc95da6823171b2362]) successfully deleted
W20260809 16:39:48.538403 4431 connection.cc:535] client connection to 127.28.62.129:37471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 16:39:48.538810 5063 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:48.538686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:37471 (local address 127.0.0.1:41532)
0809 16:39:48.538743 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:48.538771 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:37471: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 16:39:48.538975 4431 catalog_manager.cc:4584] TS 8c985aba99f8496d84710ebd3f5a3735 (127.28.62.129:37471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 318813bfb6a24d23998eaad9bad451d4: Network error: Client connection negotiation failed: client connection to 127.28.62.129:37471: connect: Connection reset by peer (error 104)
I20260809 16:39:48.539122 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 4657
I20260809 16:39:48.549361 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 4812
I20260809 16:39:48.558674 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 4081
I20260809 16:39:48.565472 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 4294
2026-08-09T16:39:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5139 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:48 dist-test-slave-wl2h krb5kdc[5101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:48 dist-test-slave-wl2h krb5kdc[5101](info): set up 2 sockets
Aug 09 16:39:48 dist-test-slave-wl2h krb5kdc[5101](info): commencing operation
krb5kdc: starting...
W20260809 16:39:49.750821 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:49 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab.
Entry for principal oryx/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab.
I20260809 16:39:49.882784 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45217
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab
--principal=oryx/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:49.968218 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:49.968466 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:49.968492 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:49.969668 5116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:49.969713 5116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 16:39:49.969727 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:49.969740 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:49.969753 5116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:49.969764 5116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:49.971397 5116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:45217
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.190.keytab
--principal=oryx/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:49.971668 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:49.971938 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:49.977536 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:49.977550 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:49.977658 5116 server_base.cc:1034] running on GCE node
W20260809 16:39:49.977550 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:49.978080 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:49.978416 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:49.979607 5116 hybrid_clock.cc:648] HybridClock initialized: now 1786293589979591 us; error 47 us; skew 500 ppm
Aug 09 16:39:49 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:49.982312 5116 init.cc:377] Logged in from keytab as oryx/127.28.62.190@KRBTEST.COM (short username oryx)
I20260809 16:39:49.982731 5116 webserver.cc:460] Webserver started at http://127.28.62.190:41007/ using document root <none> and password file <none>
I20260809 16:39:49.983003 5116 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:49.983101 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:49.983250 5116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:49.984401 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "320833cea4e844078996cf859117e170"
format_stamp: "Formatted at 2026-08-09 16:39:49 on dist-test-slave-wl2h"
server_key: "ec8ed16edeb61954a0f90f0073c5a2be"
server_key_iv: "393c7656ab47485c2cd956fde78a8d12"
server_key_version: "encryptionkey@0"
I20260809 16:39:49.984788 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "320833cea4e844078996cf859117e170"
format_stamp: "Formatted at 2026-08-09 16:39:49 on dist-test-slave-wl2h"
server_key: "ec8ed16edeb61954a0f90f0073c5a2be"
server_key_iv: "393c7656ab47485c2cd956fde78a8d12"
server_key_version: "encryptionkey@0"
I20260809 16:39:49.986414 5116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:49.987344 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:49.987525 5116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:49.987636 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "320833cea4e844078996cf859117e170"
format_stamp: "Formatted at 2026-08-09 16:39:49 on dist-test-slave-wl2h"
server_key: "ec8ed16edeb61954a0f90f0073c5a2be"
server_key_iv: "393c7656ab47485c2cd956fde78a8d12"
server_key_version: "encryptionkey@0"
I20260809 16:39:49.987727 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:50.002770 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:50.005587 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:50.005820 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:50.011129 5116 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:45217
I20260809 16:39:50.011328 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:45217 every 8 connection(s)
I20260809 16:39:50.011595 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:50.012348 5207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.014700 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Bootstrap starting.
I20260809 16:39:50.015414 5207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.015677 5207 log.cc:824] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:50.016403 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: No bootstrap required, opened a new log
I20260809 16:39:50.018808 5207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } }
I20260809 16:39:50.018998 5207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.019099 5207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320833cea4e844078996cf859117e170, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.019261 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 5116
I20260809 16:39:50.019250 5207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } }
I20260809 16:39:50.019337 5207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:50.019346 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:50.019361 5207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:50.019392 5207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:50.019508 28922 external_mini_cluster.cc:1428] Setting key c6a4fb44f49c337e8ad3252a59ef8894
I20260809 16:39:50.020267 5207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } }
I20260809 16:39:50.020398 5207 leader_election.cc:302] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320833cea4e844078996cf859117e170; no voters:
I20260809 16:39:50.020610 5207 leader_election.cc:288] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:50.020640 5213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:50.020920 5207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:50.020990 5213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [term 1 LEADER]: Becoming Leader. State: Replica: 320833cea4e844078996cf859117e170, State: Running, Role: LEADER
I20260809 16:39:50.021166 5213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } }
I20260809 16:39:50.021588 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320833cea4e844078996cf859117e170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } } }
I20260809 16:39:50.021683 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:50.021951 5215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320833cea4e844078996cf859117e170. Latest consensus state: current_term: 1 leader_uuid: "320833cea4e844078996cf859117e170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320833cea4e844078996cf859117e170" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45217 } } }
I20260809 16:39:50.022073 5215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:50.022310 5223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:50.022862 5223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:50.024639 5223 catalog_manager.cc:1347] Generated new cluster ID: f02e554799734155850c39a4ab6eab9d
I20260809 16:39:50.024709 5223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:50.034173 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.020079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.28.62.190:45217)
0809 16:39:50.020375 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.020386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.020430 (+ 44us) server_negotiation.cc:407] Connection header received
0809 16:39:50.020619 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.020624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.020709 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.021679 (+ 970us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.021793 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.023415 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.024489 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.025137 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.025269 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.028366 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.028372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.028372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.028387 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.031433 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.032027 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.032034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.032035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.032312 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.032780 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.032782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.032782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.033317 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.033645 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.033756 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.033933 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260809 16:39:50.038892 5223 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:50.039451 5223 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:50.047145 5223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Generated new TSK 0
I20260809 16:39:50.047416 5223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
I20260809 16:39:50.080530 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45217
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:50.179390 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:50.179699 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:50.179749 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:50.181211 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:50.181277 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:50.181316 5236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:50.181351 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:50.183064 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:50.183344 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:50.183579 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:50.189627 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.189815 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.189632 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:50.191629 5236 server_base.cc:1034] running on GCE node
I20260809 16:39:50.191927 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:50.192291 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:50.193454 5236 hybrid_clock.cc:648] HybridClock initialized: now 1786293590193442 us; error 47 us; skew 500 ppm
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:50.196666 5236 init.cc:377] Logged in from keytab as oryx/127.28.62.129@KRBTEST.COM (short username oryx)
I20260809 16:39:50.197358 5236 webserver.cc:460] Webserver started at http://127.28.62.129:38819/ using document root <none> and password file <none>
I20260809 16:39:50.197683 5236 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:50.197767 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:50.197935 5236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:50.199610 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "8ba8c5ca19e34a3f8d3fbe004070279a"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9c796ab307ec91e7dcb6d82b9dfde497"
server_key_iv: "8eba742c040ab142fcdb64073d7fafdc"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.200081 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "8ba8c5ca19e34a3f8d3fbe004070279a"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9c796ab307ec91e7dcb6d82b9dfde497"
server_key_iv: "8eba742c040ab142fcdb64073d7fafdc"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.202153 5236 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:50.203704 5253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.203984 5236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:50.204130 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8ba8c5ca19e34a3f8d3fbe004070279a"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9c796ab307ec91e7dcb6d82b9dfde497"
server_key_iv: "8eba742c040ab142fcdb64073d7fafdc"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.204221 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:50.236712 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:50.239056 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:50.239229 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:50.239809 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:50.239846 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.239871 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:50.239933 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:50.248589 5236 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:43913
I20260809 16:39:50.248673 5392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:43913 every 8 connection(s)
I20260809 16:39:50.249064 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:50.250288 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.240509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:34921 (local address 127.28.62.190:45217)
0809 16:39:50.240569 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.240572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.240805 (+ 233us) server_negotiation.cc:407] Connection header received
0809 16:39:50.241399 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.241400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.241412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.241504 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.242534 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.243298 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.244314 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.244538 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.244630 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.247206 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.247208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.247208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.247216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.248949 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.249474 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.249475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.249476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.249543 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.249768 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.249768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.249769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.249876 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.249928 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.250133 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.250224 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:50.250422 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.240504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.129:34921)
0809 16:39:50.240678 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.240715 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.241294 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.241649 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.241655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.242098 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.242416 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.242424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.243401 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.243402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.244220 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.244222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.244658 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.244659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.244780 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.245307 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.245317 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.247121 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.249028 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.249034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.249036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.249368 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.249606 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.249607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.249607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.249705 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.249917 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.249919 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.250028 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.250119 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.250240 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260809 16:39:50.250985 5393 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45217
I20260809 16:39:50.251108 5393 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:50.251336 5393 heartbeater.cc:511] Master 127.28.62.190:45217 requested a full tablet report, sending...
I20260809 16:39:50.251853 5150 ts_manager.cc:194] Registered new tserver with Master: 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913)
I20260809 16:39:50.252728 5150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:34921
I20260809 16:39:50.257817 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.252155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.129:56013)
0809 16:39:50.252304 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.252315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.252369 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.252496 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.252496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.252653 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.252707 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.252708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.253517 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.253517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.254033 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.254034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.254385 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.254386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.254430 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.254863 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.254867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.255185 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.256824 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.256825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.256825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.256988 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.257278 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.257278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.257279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.257337 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.257572 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.257572 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.257609 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.257664 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.257742 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":7,"thread_start_us":83,"threads_started":1}
I20260809 16:39:50.257818 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.252084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:56013 (local address 127.28.62.190:45217)
0809 16:39:50.252121 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.252122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.252405 (+ 283us) server_negotiation.cc:407] Connection header received
0809 16:39:50.252428 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.252429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.252439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.252493 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.252775 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.253421 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.254116 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.254316 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.254373 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.255296 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.255297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.255298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.255304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.256736 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.257090 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.257092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.257092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.257193 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.257403 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.257403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.257404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.257514 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.257571 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.257683 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.257754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 16:39:50.258831 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 5236
I20260809 16:39:50.258975 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:50.259251 28922 external_mini_cluster.cc:1428] Setting key b65340992dc6bbcdf69cf201b7d7cebd
WARNING: no policy specified for oryx/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
I20260809 16:39:50.288542 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45217
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:50.373415 5404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:50.373683 5404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:50.373726 5404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:50.374969 5404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:50.375094 5404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:50.375164 5404 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:50.375222 5404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:50.376942 5404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:50.377233 5404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:50.377485 5404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:50.382665 5412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.382665 5410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.382789 5409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:50.382856 5404 server_base.cc:1034] running on GCE node
I20260809 16:39:50.383172 5404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:50.383392 5404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:50.384536 5404 hybrid_clock.cc:648] HybridClock initialized: now 1786293590384513 us; error 39 us; skew 500 ppm
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:50.387195 5404 init.cc:377] Logged in from keytab as oryx/127.28.62.130@KRBTEST.COM (short username oryx)
I20260809 16:39:50.387594 5404 webserver.cc:460] Webserver started at http://127.28.62.130:43643/ using document root <none> and password file <none>
I20260809 16:39:50.387852 5404 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:50.387923 5404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:50.388079 5404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:50.389142 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "57fe5118aff94b6780e97e5220fe5dbe"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "b30023b5ca22c8e527fd070f9f216505"
server_key_iv: "6e628e62859fa14316146cf88b3a0ac1"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.389511 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "57fe5118aff94b6780e97e5220fe5dbe"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "b30023b5ca22c8e527fd070f9f216505"
server_key_iv: "6e628e62859fa14316146cf88b3a0ac1"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.391216 5404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:50.392169 5419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.392503 5404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:50.392599 5404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "57fe5118aff94b6780e97e5220fe5dbe"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "b30023b5ca22c8e527fd070f9f216505"
server_key_iv: "6e628e62859fa14316146cf88b3a0ac1"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.392695 5404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:50.404646 5404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:50.406884 5404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:50.407137 5404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:50.407800 5404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:50.407871 5404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.407918 5404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:50.407946 5404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.414465 5404 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:44327
I20260809 16:39:50.414641 5550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:44327 every 8 connection(s)
I20260809 16:39:50.414893 5404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:50.418253 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.408377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:50125 (local address 127.28.62.190:45217)
0809 16:39:50.408448 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.408450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.408650 (+ 200us) server_negotiation.cc:407] Connection header received
0809 16:39:50.409236 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.409238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.409251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.409316 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.410220 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.411091 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.412747 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.412960 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.413054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.415733 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.415735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.415735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.415741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.416937 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.417323 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.417323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.417324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.417377 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.417637 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.417639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.417639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.417768 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.417846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.418086 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.418187 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 16:39:50.418457 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.408362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.130:50125)
0809 16:39:50.408529 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.408565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.409123 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.409385 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.409390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.409796 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.410090 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.410096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.411190 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.411191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.412650 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.412653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.413068 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.413069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.413214 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.413869 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.413880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.415645 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.417009 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.417015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.417016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.417247 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.417444 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.417445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.417445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.417542 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.417864 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.417869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.417984 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.418070 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.418192 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260809 16:39:50.418982 5551 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45217
I20260809 16:39:50.419113 5551 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:50.419305 5551 heartbeater.cc:511] Master 127.28.62.190:45217 requested a full tablet report, sending...
I20260809 16:39:50.419766 5149 ts_manager.cc:194] Registered new tserver with Master: 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327)
I20260809 16:39:50.420303 5149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:50125
I20260809 16:39:50.424830 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 5404
I20260809 16:39:50.424981 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:50.425225 28922 external_mini_cluster.cc:1428] Setting key 992a099fe008e2cf0dd72d25b50b4f2f
I20260809 16:39:50.426400 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.420258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:42263 (local address 127.28.62.190:45217)
0809 16:39:50.420297 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.420299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.420625 (+ 326us) server_negotiation.cc:407] Connection header received
0809 16:39:50.420651 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.420653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.420663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.420719 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.421119 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.421783 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.422564 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.422749 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.422806 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.423790 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.423791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.423792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.423798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.425150 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.425534 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.425535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.425535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.425603 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.425825 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.425826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.425826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.425967 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.426044 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.426220 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.426309 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:39:50.426411 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.420104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.130:42263)
0809 16:39:50.420514 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.420529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.420588 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.420734 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.420735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.420937 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.420997 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.420999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.421881 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.421882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.422471 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.422473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.422819 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.422819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.422870 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.423394 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.423398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.423678 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.425229 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.425230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.425231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.425455 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.425677 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.425678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.425678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.425750 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.426036 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.426036 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.426083 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.426187 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.426302 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
I20260809 16:39:50.454706 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45217
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:50.539752 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:50.539978 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:50.540004 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:50.541180 5561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:50.541226 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:50.541249 5561 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 16:39:50.541308 5561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:50.542896 5561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.62.190:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:50.543161 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:50.543423 5561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:50.548527 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.548538 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:50.548538 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:50.548938 5561 server_base.cc:1034] running on GCE node
I20260809 16:39:50.549155 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:50.549396 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:50.550580 5561 hybrid_clock.cc:648] HybridClock initialized: now 1786293590550546 us; error 32 us; skew 500 ppm
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:50.553454 5561 init.cc:377] Logged in from keytab as oryx/127.28.62.131@KRBTEST.COM (short username oryx)
I20260809 16:39:50.553928 5561 webserver.cc:460] Webserver started at http://127.28.62.131:37805/ using document root <none> and password file <none>
I20260809 16:39:50.554169 5561 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:50.554215 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:50.554430 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:50.555596 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "07f7382d83624398a3b1559203ce6381"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9d281b31b18cb617e99f1710e9562379"
server_key_iv: "53b1a54fe00c369ec03943cc70b2cb2c"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.556006 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "07f7382d83624398a3b1559203ce6381"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9d281b31b18cb617e99f1710e9562379"
server_key_iv: "53b1a54fe00c369ec03943cc70b2cb2c"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.557472 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:50.558389 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.558686 5561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:50.558816 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "07f7382d83624398a3b1559203ce6381"
format_stamp: "Formatted at 2026-08-09 16:39:50 on dist-test-slave-wl2h"
server_key: "9d281b31b18cb617e99f1710e9562379"
server_key_iv: "53b1a54fe00c369ec03943cc70b2cb2c"
server_key_version: "encryptionkey@0"
I20260809 16:39:50.558897 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:50.573769 5561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:50.576293 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:50.576462 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:50.577159 5561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:50.577211 5561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.577256 5561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:50.577301 5561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:50.584964 5561 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:41963
I20260809 16:39:50.585041 5720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:41963 every 8 connection(s)
I20260809 16:39:50.585399 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:50.588083 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.577612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:33179 (local address 127.28.62.190:45217)
0809 16:39:50.577690 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.577693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.577936 (+ 243us) server_negotiation.cc:407] Connection header received
0809 16:39:50.578466 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.578468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.578483 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.578572 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.579383 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.579944 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.581145 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.581415 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.581503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.584923 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.584924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.584925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.584934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.586585 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.587084 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.587085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.587085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.587149 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.587423 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.587424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.587424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.587547 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.587622 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.587939 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.588016 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:39:50.588233 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.577607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.131:33179)
0809 16:39:50.577787 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.577833 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.578362 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.578637 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.578640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.578905 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.579269 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.579276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.580055 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.580056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.581045 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.581048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.581513 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.581514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.581646 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.582333 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.582346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.584804 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.586731 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.586738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.586741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.586977 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.587232 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.587233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.587233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.587345 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.587626 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.587633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.587774 (+ 141us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.587892 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.588040 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260809 16:39:50.588757 5721 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45217
I20260809 16:39:50.588833 5721 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:50.589033 5721 heartbeater.cc:511] Master 127.28.62.190:45217 requested a full tablet report, sending...
I20260809 16:39:50.589465 5150 ts_manager.cc:194] Registered new tserver with Master: 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:50.589991 5150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:33179
I20260809 16:39:50.591001 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 5561
I20260809 16:39:50.591173 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:39:50.591382 28922 external_mini_cluster.cc:1428] Setting key b702311b9ba69c3dc3b53d3ac37c0953
I20260809 16:39:50.592355 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:39:50.596710 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.589913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:34819 (local address 127.28.62.190:45217)
0809 16:39:50.589963 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.589965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.590025 (+ 60us) server_negotiation.cc:407] Connection header received
0809 16:39:50.590147 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.590147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.590158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.590234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.590625 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.591348 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.592437 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.592669 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.592725 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.593765 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.593766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.593766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.593773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.595422 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.595922 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.595923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.595923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.595980 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.596220 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.596220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.596220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.596324 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.596380 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.596588 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.596648 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 16:39:50.596859 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.589841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.28.62.131:34819)
0809 16:39:50.589968 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.589979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.590046 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.590280 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.590281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.590481 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:50.590534 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.590535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.591459 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.591460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.592354 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.592357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.592746 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.592747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.592816 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.593250 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:50.593256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:50.593671 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:50.595510 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.595511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.595511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.595831 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.596062 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:50.596063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:50.596063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:50.596147 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:50.596400 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:50.596402 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:50.596462 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.596554 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.596690 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 16:39:50.617036 5212 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:50.611645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.28.62.190:45217)
0809 16:39:50.611728 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.611730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.611740 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:50.611768 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.611769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.611779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.611845 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.612410 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.612913 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.613619 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.613802 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.613859 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.616935 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43398: BlockingRecv error: recv got EOF from 127.0.0.1:43398 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 16:39:50 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:50.624770 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.617866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.28.62.190:45217)
0809 16:39:50.617910 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.617913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.618034 (+ 121us) server_negotiation.cc:407] Connection header received
0809 16:39:50.618072 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.618073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.618106 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.618159 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:50.618630 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.619194 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.619878 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.620064 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.620138 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.622368 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:50.622369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:50.622369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:50.622376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:50.623736 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.624061 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.624062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.624062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.624111 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:50.624324 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:50.624324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:50.624325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:50.624426 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:50.624501 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.624642 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.624700 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:39:50.627480 5150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:50.629094 5150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:50.636718 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.631778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:41963 (local address 127.0.0.1:58294)
0809 16:39:50.632152 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.632199 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.632297 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.632639 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.632642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.632644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:50.632720 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.632722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.634068 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.634069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.635612 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.635613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.636384 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.636385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.636478 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.636494 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.636560 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.636624 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":36,"thread_start_us":77,"threads_started":1}
I20260809 16:39:50.636833 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.631898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.28.62.129:43913)
0809 16:39:50.632093 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.632097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.632568 (+ 471us) server_negotiation.cc:407] Connection header received
0809 16:39:50.632705 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.632708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.632726 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.632782 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:50.633002 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.634105 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.635709 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.636469 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.636537 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.636619 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.636686 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.636718 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260809 16:39:50.637001 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.631789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43913 (local address 127.0.0.1:39546)
0809 16:39:50.632476 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.632487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.632603 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.632903 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.632903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.632905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:50.632938 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.632940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.634299 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.634300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.635610 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.635613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.636554 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.636555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.636622 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.636626 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.636675 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.636716 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260809 16:39:50.637010 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.631870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.28.62.131:41963)
0809 16:39:50.632079 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.632084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.632275 (+ 191us) server_negotiation.cc:407] Connection header received
0809 16:39:50.632460 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.632463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.632481 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.632554 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:50.632807 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.633966 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.635726 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.636295 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.636372 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.636450 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.636876 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.636919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260809 16:39:50.637701 5308 tablet_service.cc:1467] Processing CreateTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 (DEFAULT_TABLE table=test-table [id=8910ab19d4154b03842305e7a87fcd36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.637706 5640 tablet_service.cc:1467] Processing CreateTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 (DEFAULT_TABLE table=test-table [id=8910ab19d4154b03842305e7a87fcd36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.638048 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2bf76a522d4ef29961e6813d0c3430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.638058 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2bf76a522d4ef29961e6813d0c3430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.639711 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.632002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.0.0.1:59444)
0809 16:39:50.632543 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.632554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.632606 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.632879 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.632880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.632882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:50.632939 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.632940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.635236 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.635237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.636149 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.636152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.639440 (+ 3288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.639441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.639516 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.639524 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.639598 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.639641 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":88,"threads_started":1}
I20260809 16:39:50.639788 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.632240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.28.62.130:44327)
0809 16:39:50.632455 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.632460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.632679 (+ 219us) server_negotiation.cc:407] Connection header received
0809 16:39:50.632726 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.632729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.632746 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.632867 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:50.634200 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.635143 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.636250 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.639337 (+ 3087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.639433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.639521 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.639615 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.639668 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":142,"threads_started":1}
I20260809 16:39:50.640846 5452 tablet_service.cc:1467] Processing CreateTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 (DEFAULT_TABLE table=test-table [id=8910ab19d4154b03842305e7a87fcd36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.641199 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2bf76a522d4ef29961e6813d0c3430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.641399 5754 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Bootstrap starting.
I20260809 16:39:50.642103 5754 tablet_bootstrap.cc:654] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.641986 5755 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Bootstrap starting.
I20260809 16:39:50.642510 5754 log.cc:824] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:50.642812 5755 tablet_bootstrap.cc:654] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.643236 5755 log.cc:824] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:50.643419 5754 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: No bootstrap required, opened a new log
I20260809 16:39:50.643500 5754 ts_tablet_manager.cc:1397] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:50.644835 5758 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Bootstrap starting.
I20260809 16:39:50.644982 5754 raft_consensus.cc:348] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.645171 5754 raft_consensus.cc:374] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.645210 5754 raft_consensus.cc:729] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.645335 5754 consensus_queue.cc:260] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.645478 5758 tablet_bootstrap.cc:654] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.645656 5754 ts_tablet_manager.cc:1428] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:50.645816 5721 heartbeater.cc:503] Master 127.28.62.190:45217 was elected leader, sending a full tablet report...
I20260809 16:39:50.645826 5758 log.cc:824] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:50.646545 5755 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: No bootstrap required, opened a new log
I20260809 16:39:50.646634 5755 ts_tablet_manager.cc:1397] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 16:39:50.647405 5758 tablet_bootstrap.cc:492] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: No bootstrap required, opened a new log
I20260809 16:39:50.647467 5758 ts_tablet_manager.cc:1397] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:50.648232 5755 raft_consensus.cc:348] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.648393 5755 raft_consensus.cc:374] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.648440 5755 raft_consensus.cc:729] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.648587 5755 consensus_queue.cc:260] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.648900 5755 ts_tablet_manager.cc:1428] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:39:50.648948 5393 heartbeater.cc:503] Master 127.28.62.190:45217 was elected leader, sending a full tablet report...
I20260809 16:39:50.649286 5758 raft_consensus.cc:348] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.649485 5758 raft_consensus.cc:374] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.649523 5758 raft_consensus.cc:729] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.649657 5758 consensus_queue.cc:260] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.649963 5758 ts_tablet_manager.cc:1428] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:39:50.650054 5551 heartbeater.cc:503] Master 127.28.62.190:45217 was elected leader, sending a full tablet report...
W20260809 16:39:50.665624 5552 tablet.cc:2369] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:50.697018 5761 raft_consensus.cc:482] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:50.697160 5761 raft_consensus.cc:504] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.697472 5761 leader_election.cc:288] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963), 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327)
I20260809 16:39:50.701637 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.697867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47589 (local address 127.28.62.131:41963)
0809 16:39:50.697936 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.697938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.697950 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:50.698013 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.698014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.698024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.698078 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:50.698317 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.699169 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.700631 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.701207 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.701291 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.701301 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.701522 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.701564 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 16:39:50.701637 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.697988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:57431 (local address 127.28.62.130:44327)
0809 16:39:50.698037 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.698039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.698173 (+ 134us) server_negotiation.cc:407] Connection header received
0809 16:39:50.698298 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.698299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.698311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.698386 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:50.698519 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.699237 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.700631 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.701245 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.701310 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.701319 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.701522 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.701564 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 16:39:50.701668 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.697899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.28.62.129:57431)
0809 16:39:50.698083 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.698093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.698148 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.698396 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.698396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.698398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:50.698451 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.698452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.699310 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.699311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.700530 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.700532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.701327 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.701328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.701429 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.701433 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.701497 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.701554 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":45,"thread_start_us":120,"threads_started":1}
I20260809 16:39:50.702085 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2bf76a522d4ef29961e6813d0c3430" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe" is_pre_election: true
I20260809 16:39:50.702232 5502 raft_consensus.cc:2454] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:50.701680 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.697765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:41963 (local address 127.28.62.129:47589)
0809 16:39:50.697857 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:50.697873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:50.697942 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:50.698137 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:50.698138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:50.698140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:50.698214 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.698216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.699249 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.699249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.700528 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:50.700532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.701327 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.701328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:50.701433 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.701434 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:50.701497 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 16:39:50.701554 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":11}
I20260809 16:39:50.702564 5254 leader_election.cc:302] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fe5118aff94b6780e97e5220fe5dbe, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:50.702698 5761 raft_consensus.cc:2781] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:50.702760 5761 raft_consensus.cc:482] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:50.702778 5761 raft_consensus.cc:3037] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:50.702745 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2bf76a522d4ef29961e6813d0c3430" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381" is_pre_election: true
I20260809 16:39:50.702960 5663 raft_consensus.cc:2454] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:50.703642 5761 raft_consensus.cc:504] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.703765 5761 leader_election.cc:288] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Requested vote from peers 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963), 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327)
I20260809 16:39:50.703946 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2bf76a522d4ef29961e6813d0c3430" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381"
I20260809 16:39:50.703989 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2bf76a522d4ef29961e6813d0c3430" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe"
I20260809 16:39:50.704035 5663 raft_consensus.cc:3037] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:50.704067 5502 raft_consensus.cc:3037] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:50.705000 5663 raft_consensus.cc:2454] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:50.705000 5502 raft_consensus.cc:2454] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:50.705260 5254 leader_election.cc:302] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fe5118aff94b6780e97e5220fe5dbe, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:50.705363 5761 raft_consensus.cc:2781] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:50.705538 5761 raft_consensus.cc:686] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Becoming Leader. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Running, Role: LEADER
I20260809 16:39:50.705648 5761 consensus_queue.cc:237] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } }
I20260809 16:39:50.706635 5149 catalog_manager.cc:5495] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129). New cstate: current_term: 1 leader_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } health_report { overall_health: HEALTHY } } }
I20260809 16:39:50.731048 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:50.728559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.28.62.129:43913)
0809 16:39:50.728628 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:50.728630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:50.728639 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:50.728678 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:50.728679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:50.728697 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:50.728781 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:50.728903 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.729394 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.730253 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:50.730433 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:50.730484 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:50.730670 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:50.730772 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:50.730830 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:50.730927 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:39:50.730945 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:39:50.733597 5502 raft_consensus.cc:1261] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:50.733590 5663 raft_consensus.cc:1261] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:50.734046 5766 consensus_queue.cc:1035] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:50.734213 5761 consensus_queue.cc:1035] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:50.737485 5771 mvcc.cc:205] Tried to move back new op lower bound from 7316658547641131008 to 7316658547531116544. Current Snapshot: MvccSnapshot[applied={T|T < 7316658547641131008}]
I20260809 16:39:50.737696 5770 mvcc.cc:205] Tried to move back new op lower bound from 7316658547641131008 to 7316658547531116544. Current Snapshot: MvccSnapshot[applied={T|T < 7316658547641131008}]
I20260809 16:39:50.739101 5149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:50.739200 5149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:50.740023 5149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Sending DeleteTablet for 3 replicas of tablet ab2bf76a522d4ef29961e6813d0c3430
I20260809 16:39:50.740336 5640 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:50 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260809 16:39:50.740375 5452 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:50 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:59444
I20260809 16:39:50.740376 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2bf76a522d4ef29961e6813d0c3430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:50 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:39546
I20260809 16:39:50.740777 5779 tablet_replica.cc:330] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: stopping tablet replica
I20260809 16:39:50.740892 5779 raft_consensus.cc:2229] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:50.740841 5149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:50.740983 5149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:50.741276 5779 raft_consensus.cc:2258] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:50.741482 5781 tablet_replica.cc:330] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: stopping tablet replica
I20260809 16:39:50.741596 5781 raft_consensus.cc:2229] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:50.741714 5781 raft_consensus.cc:2258] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:50.741981 5779 ts_tablet_manager.cc:1905] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:50.742100 5780 tablet_replica.cc:330] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: stopping tablet replica
I20260809 16:39:50.742210 5780 raft_consensus.cc:2229] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:50.742318 5780 raft_consensus.cc:2258] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:50.742506 5452 tablet_service.cc:1467] Processing CreateTablet for tablet 931f8edfde3d4176845798f9085a9b63 (DEFAULT_TABLE table=test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.742515 5640 tablet_service.cc:1467] Processing CreateTablet for tablet 931f8edfde3d4176845798f9085a9b63 (DEFAULT_TABLE table=test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.742632 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931f8edfde3d4176845798f9085a9b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.742646 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931f8edfde3d4176845798f9085a9b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.744065 5758 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Bootstrap starting.
I20260809 16:39:50.744079 5779 ts_tablet_manager.cc:1918] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:50.744136 5754 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Bootstrap starting.
I20260809 16:39:50.744277 5779 log.cc:1196] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/ab2bf76a522d4ef29961e6813d0c3430
I20260809 16:39:50.744535 5758 tablet_bootstrap.cc:654] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.744733 5754 tablet_bootstrap.cc:654] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.744755 5780 ts_tablet_manager.cc:1905] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:50.744979 5779 ts_tablet_manager.cc:1939] T ab2bf76a522d4ef29961e6813d0c3430 P 07f7382d83624398a3b1559203ce6381: Deleting consensus metadata
I20260809 16:39:50.745121 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 931f8edfde3d4176845798f9085a9b63 (DEFAULT_TABLE table=test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:50.745249 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931f8edfde3d4176845798f9085a9b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:50.745401 5781 ts_tablet_manager.cc:1905] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:50.745571 5134 catalog_manager.cc:4841] TS 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963): tablet ab2bf76a522d4ef29961e6813d0c3430 (table test-table [id=8910ab19d4154b03842305e7a87fcd36]) successfully deleted
I20260809 16:39:50.746377 5755 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Bootstrap starting.
I20260809 16:39:50.746728 5780 ts_tablet_manager.cc:1918] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:50.746796 5780 log.cc:1196] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/ab2bf76a522d4ef29961e6813d0c3430
I20260809 16:39:50.746811 5755 tablet_bootstrap.cc:654] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:50.747188 5780 ts_tablet_manager.cc:1939] T ab2bf76a522d4ef29961e6813d0c3430 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting consensus metadata
I20260809 16:39:50.747293 5781 ts_tablet_manager.cc:1918] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:50.747375 5781 log.cc:1196] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/ab2bf76a522d4ef29961e6813d0c3430
I20260809 16:39:50.747478 5755 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: No bootstrap required, opened a new log
I20260809 16:39:50.747516 5755 ts_tablet_manager.cc:1397] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:50.747674 5134 catalog_manager.cc:4841] TS 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327): tablet ab2bf76a522d4ef29961e6813d0c3430 (table test-table [id=8910ab19d4154b03842305e7a87fcd36]) successfully deleted
I20260809 16:39:50.747691 5781 ts_tablet_manager.cc:1939] T ab2bf76a522d4ef29961e6813d0c3430 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting consensus metadata
I20260809 16:39:50.747673 5755 raft_consensus.cc:348] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.747747 5755 raft_consensus.cc:374] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.747773 5755 raft_consensus.cc:729] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.747829 5755 consensus_queue.cc:260] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.747965 5755 ts_tablet_manager.cc:1428] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:50.747999 5758 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: No bootstrap required, opened a new log
I20260809 16:39:50.748066 5758 ts_tablet_manager.cc:1397] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 16:39:50.748234 5758 raft_consensus.cc:348] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.748310 5758 raft_consensus.cc:374] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.748337 5758 raft_consensus.cc:729] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.748378 5758 consensus_queue.cc:260] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.748471 5758 ts_tablet_manager.cc:1428] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:50.748734 5137 catalog_manager.cc:4841] TS 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913): tablet ab2bf76a522d4ef29961e6813d0c3430 (table test-table [id=8910ab19d4154b03842305e7a87fcd36]) successfully deleted
I20260809 16:39:50.749322 5754 tablet_bootstrap.cc:492] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: No bootstrap required, opened a new log
I20260809 16:39:50.749379 5754 ts_tablet_manager.cc:1397] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260809 16:39:50.749526 5754 raft_consensus.cc:348] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.749593 5754 raft_consensus.cc:374] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:50.749606 5754 raft_consensus.cc:729] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Initialized, Role: FOLLOWER
I20260809 16:39:50.749660 5754 consensus_queue.cc:260] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:50.749753 5754 ts_tablet_manager.cc:1428] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260809 16:39:50.749852 5394 tablet.cc:2369] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:39:50.836018 5722 tablet.cc:2369] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:39:51.088810 5774 raft_consensus.cc:482] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:51.088951 5774 raft_consensus.cc:504] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.089167 5774 leader_election.cc:288] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.089453 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931f8edfde3d4176845798f9085a9b63" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe" is_pre_election: true
I20260809 16:39:51.089483 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931f8edfde3d4176845798f9085a9b63" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381" is_pre_election: true
I20260809 16:39:51.089592 5502 raft_consensus.cc:2454] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:51.089587 5663 raft_consensus.cc:2454] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:51.089771 5254 leader_election.cc:302] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7382d83624398a3b1559203ce6381, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.089936 5774 raft_consensus.cc:2781] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:51.090009 5774 raft_consensus.cc:482] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:51.090034 5774 raft_consensus.cc:3037] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.090947 5774 raft_consensus.cc:504] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.091130 5774 leader_election.cc:288] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Requested vote from peers 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.091357 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931f8edfde3d4176845798f9085a9b63" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe"
I20260809 16:39:51.091411 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931f8edfde3d4176845798f9085a9b63" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381"
I20260809 16:39:51.091490 5663 raft_consensus.cc:3037] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.091511 5502 raft_consensus.cc:3037] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.092191 5663 raft_consensus.cc:2454] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:51.092239 5502 raft_consensus.cc:2454] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:51.092424 5254 leader_election.cc:302] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fe5118aff94b6780e97e5220fe5dbe, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.092532 5774 raft_consensus.cc:2781] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:51.092599 5774 raft_consensus.cc:686] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Becoming Leader. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Running, Role: LEADER
I20260809 16:39:51.092656 5774 consensus_queue.cc:237] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.093207 5149 catalog_manager.cc:5495] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129). New cstate: current_term: 1 leader_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } health_report { overall_health: UNKNOWN } } }
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.190@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:51.131618 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.123836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45217 (local address 127.0.0.1:43416)
0809 16:39:51.124028 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.124042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.124175 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.124340 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.124340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.124664 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:51.124718 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.124721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.125407 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.125407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.125931 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.125932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.126313 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.126314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.126370 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.126969 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:51.126975 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:51.128838 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:51.130438 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:51.130444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:51.130446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:51.130657 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:51.130856 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:51.130856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:51.130857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:51.130912 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:51.131214 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:51.131249 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:51.131302 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.131380 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.131493 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260809 16:39:51.131664 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.123907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.28.62.190:45217)
0809 16:39:51.123999 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.124001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.124116 (+ 115us) server_negotiation.cc:407] Connection header received
0809 16:39:51.124247 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.124248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.124260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.124335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:51.124809 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.125311 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.126016 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.126206 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.126298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.128944 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:51.128946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:51.128946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:51.128955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:51.130346 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:51.130726 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:51.130727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:51.130727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:51.130795 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:51.130965 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:51.130965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:51.130966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:51.131140 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:51.131196 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.131409 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.131542 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":60}
I20260809 16:39:51.133970 5149 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:43416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 16:39:51.134253 5149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:51.135927 5640 tablet_service.cc:1467] Processing CreateTablet for tablet 94a1467f60b545bc8dcd7ed7c9965063 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d7baff64688409d826c424e2a6d7313]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:51.135962 5452 tablet_service.cc:1467] Processing CreateTablet for tablet 94a1467f60b545bc8dcd7ed7c9965063 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d7baff64688409d826c424e2a6d7313]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:51.135948 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 94a1467f60b545bc8dcd7ed7c9965063 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d7baff64688409d826c424e2a6d7313]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 16:39:51.136111 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a1467f60b545bc8dcd7ed7c9965063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.136116 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a1467f60b545bc8dcd7ed7c9965063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.136118 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a1467f60b545bc8dcd7ed7c9965063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.137375 5754 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: Bootstrap starting.
I20260809 16:39:51.137785 5754 tablet_bootstrap.cc:654] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.137794 5758 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: Bootstrap starting.
I20260809 16:39:51.138201 5758 tablet_bootstrap.cc:654] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.138561 5755 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: Bootstrap starting.
I20260809 16:39:51.138586 5754 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: No bootstrap required, opened a new log
I20260809 16:39:51.138626 5754 ts_tablet_manager.cc:1397] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:51.138782 5754 raft_consensus.cc:348] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.138873 5754 raft_consensus.cc:374] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.138900 5754 raft_consensus.cc:729] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.138913 5755 tablet_bootstrap.cc:654] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.138950 5754 consensus_queue.cc:260] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.139091 5754 ts_tablet_manager.cc:1428] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:51.139263 5758 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: No bootstrap required, opened a new log
I20260809 16:39:51.139251 5759 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.139325 5758 ts_tablet_manager.cc:1397] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:51.139351 5759 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.139474 5758 raft_consensus.cc:348] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.139564 5758 raft_consensus.cc:374] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.139585 5758 raft_consensus.cc:729] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.139605 5755 tablet_bootstrap.cc:492] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: No bootstrap required, opened a new log
I20260809 16:39:51.139633 5758 consensus_queue.cc:260] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.139680 5755 ts_tablet_manager.cc:1397] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260809 16:39:51.139737 5758 ts_tablet_manager.cc:1428] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.139786 5755 raft_consensus.cc:348] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.139850 5755 raft_consensus.cc:374] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.139799 5762 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.139873 5755 raft_consensus.cc:729] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.139881 5762 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.139922 5755 consensus_queue.cc:260] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.140014 5755 ts_tablet_manager.cc:1428] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.140081 5774 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.140188 5774 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.148947 5762 raft_consensus.cc:482] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:51.149055 5762 raft_consensus.cc:504] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.149475 5762 leader_election.cc:288] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.153447 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.149750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:53877 (local address 127.28.62.129:43913)
0809 16:39:51.149793 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.149795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.149807 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:51.149881 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.149882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.149894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.149955 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:51.150113 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.150864 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.152519 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.153067 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.153148 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.153157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.153343 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.153385 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:51.153447 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.149680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:53655 (local address 127.28.62.131:41963)
0809 16:39:51.149727 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.149730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.149819 (+ 89us) server_negotiation.cc:407] Connection header received
0809 16:39:51.149881 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.149881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.149892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.149961 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:51.150113 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.150886 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.152519 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.153066 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.153148 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.153157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.153334 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.153378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:39:51.153472 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.149635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:41963 (local address 127.28.62.130:53655)
0809 16:39:51.149757 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.149768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.149825 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.149964 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.149965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.149966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:51.150039 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.150040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.150967 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.150968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.152425 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.152427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.153159 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.153159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.153233 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.153242 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.153316 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.153351 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":49,"thread_start_us":59,"threads_started":1}
I20260809 16:39:51.153472 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.149637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43913 (local address 127.28.62.130:53877)
0809 16:39:51.149768 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.149777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.149825 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.149958 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.149959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.149961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:51.150039 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.150040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.150968 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.150968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.152422 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.152426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.153157 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.153158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.153233 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.153240 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.153301 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.153351 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":103,"thread_start_us":60,"threads_started":1}
I20260809 16:39:51.153858 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a1467f60b545bc8dcd7ed7c9965063" candidate_uuid: "57fe5118aff94b6780e97e5220fe5dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381" is_pre_election: true
I20260809 16:39:51.153882 5333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a1467f60b545bc8dcd7ed7c9965063" candidate_uuid: "57fe5118aff94b6780e97e5220fe5dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" is_pre_election: true
I20260809 16:39:51.153957 5663 raft_consensus.cc:2454] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fe5118aff94b6780e97e5220fe5dbe in term 0.
I20260809 16:39:51.154000 5333 raft_consensus.cc:2454] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57fe5118aff94b6780e97e5220fe5dbe in term 0.
I20260809 16:39:51.154194 5423 leader_election.cc:302] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fe5118aff94b6780e97e5220fe5dbe, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.154367 5762 raft_consensus.cc:2781] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:51.154421 5762 raft_consensus.cc:482] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:51.154449 5762 raft_consensus.cc:3037] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.155262 5762 raft_consensus.cc:504] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.155406 5762 leader_election.cc:288] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [CANDIDATE]: Term 1 election: Requested vote from peers 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.155601 5333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a1467f60b545bc8dcd7ed7c9965063" candidate_uuid: "57fe5118aff94b6780e97e5220fe5dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a"
I20260809 16:39:51.155598 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a1467f60b545bc8dcd7ed7c9965063" candidate_uuid: "57fe5118aff94b6780e97e5220fe5dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381"
I20260809 16:39:51.155678 5333 raft_consensus.cc:3037] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.155678 5663 raft_consensus.cc:3037] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.156438 5663 raft_consensus.cc:2454] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fe5118aff94b6780e97e5220fe5dbe in term 1.
I20260809 16:39:51.156498 5333 raft_consensus.cc:2454] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57fe5118aff94b6780e97e5220fe5dbe in term 1.
I20260809 16:39:51.156744 5423 leader_election.cc:302] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fe5118aff94b6780e97e5220fe5dbe, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.156867 5762 raft_consensus.cc:2781] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:51.157029 5762 raft_consensus.cc:686] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 LEADER]: Becoming Leader. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Running, Role: LEADER
I20260809 16:39:51.157104 5762 consensus_queue.cc:237] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.157439 5803 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: TxnStatusTablet state changed. Reason: New leader 57fe5118aff94b6780e97e5220fe5dbe. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.157588 5803 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:51.157814 5806 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 16:39:51.157840 5150 catalog_manager.cc:5495] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130). New cstate: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:51.157929 5806 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 16:39:51.163844 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.161020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.0.0.1:59458)
0809 16:39:51.161107 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.161118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.161175 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.161392 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.161393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.161394 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:51.161438 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.161440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.162311 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.162312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.162868 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.162870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.163370 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.163371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.163413 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.163427 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.163703 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.163709 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.163743 (+ 34us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.163789 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 16:39:51.163846 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.161052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.28.62.130:44327)
0809 16:39:51.161090 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.161092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.161256 (+ 164us) server_negotiation.cc:407] Connection header received
0809 16:39:51.161287 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.161288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.161305 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.161368 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.161511 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.162218 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.162957 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.163284 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.163353 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.163519 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.163635 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.163713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.163758 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.163779 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:39:51.165057 5663 raft_consensus.cc:1261] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Refusing update from remote peer 57fe5118aff94b6780e97e5220fe5dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.165077 5333 raft_consensus.cc:1261] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Refusing update from remote peer 57fe5118aff94b6780e97e5220fe5dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.165367 5803 consensus_queue.cc:1035] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.165515 5762 consensus_queue.cc:1035] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.166082 5774 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: TxnStatusTablet state changed. Reason: New leader 57fe5118aff94b6780e97e5220fe5dbe. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.166181 5774 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.166510 5759 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: TxnStatusTablet state changed. Reason: New leader 57fe5118aff94b6780e97e5220fe5dbe. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.166604 5759 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.167239 5762 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.167335 5762 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:51.167384 5762 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.167423 5762 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 57fe5118aff94b6780e97e5220fe5dbe: This TxnStatusTablet replica's current role is: LEADER
I20260809 16:39:51.167745 5759 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.167841 5759 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 07f7382d83624398a3b1559203ce6381: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.168237 5774 tablet_replica.cc:439] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57fe5118aff94b6780e97e5220fe5dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } } }
I20260809 16:39:51.168339 5774 tablet_replica.cc:442] T 94a1467f60b545bc8dcd7ed7c9965063 P 8ba8c5ca19e34a3f8d3fbe004070279a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 16:39:51.174307 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.171211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:40993 (local address 127.28.62.130:44327)
0809 16:39:51.171258 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.171261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.171370 (+ 109us) server_negotiation.cc:407] Connection header received
0809 16:39:51.171438 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.171439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.171448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.171498 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.171617 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.172161 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.173480 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.173693 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.173760 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.173977 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.174008 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.174064 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.174224 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.174245 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:39:51.174659 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.171103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.28.62.129:40993)
0809 16:39:51.171296 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.171310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.171378 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.171502 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.171503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.171505 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:51.171561 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.171562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.172256 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.172258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.173367 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.173372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.173775 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.173776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.173841 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.173861 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.174075 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.174077 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.174270 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.174333 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260809 16:39:51.177426 5663 raft_consensus.cc:1261] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.177474 5502 raft_consensus.cc:1261] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.177686 5774 consensus_queue.cc:1035] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.177804 5761 consensus_queue.cc:1035] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.179538 5769 mvcc.cc:205] Tried to move back new op lower bound from 7316658549461360640 to 7316658549115858944. Current Snapshot: MvccSnapshot[applied={T|T < 7316658549461360640}]
I20260809 16:39:51.186955 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.183880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:54737 (local address 127.28.62.129:43913)
0809 16:39:51.183940 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.183942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.184150 (+ 208us) server_negotiation.cc:407] Connection header received
0809 16:39:51.184175 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.184176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.184184 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.184230 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.184402 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.184919 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.186268 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.186464 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.186523 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.186737 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.186766 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.186809 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.186885 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.186903 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:51.187067 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.183780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43913 (local address 127.28.62.130:54737)
0809 16:39:51.184021 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.184034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.184105 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.184251 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.184252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.184253 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:51.184332 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.184333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.185026 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.185027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.186176 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.186179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.186556 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.186558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.186633 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.186650 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.186825 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.186827 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.186883 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.186929 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":122,"threads_started":1}
I20260809 16:39:51.194907 5148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.195053 5148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.195772 5148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Sending DeleteTablet for 3 replicas of tablet 931f8edfde3d4176845798f9085a9b63
I20260809 16:39:51.196065 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 931f8edfde3d4176845798f9085a9b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:39546
I20260809 16:39:51.196070 5640 tablet_service.cc:1514] Processing DeleteTablet for tablet 931f8edfde3d4176845798f9085a9b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260809 16:39:51.196158 5781 tablet_replica.cc:330] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: stopping tablet replica
I20260809 16:39:51.196213 5781 raft_consensus.cc:2229] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:51.196231 5779 tablet_replica.cc:330] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: stopping tablet replica
I20260809 16:39:51.196281 5779 raft_consensus.cc:2229] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.196286 5781 raft_consensus.cc:2258] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.196316 5779 raft_consensus.cc:2258] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.196444 5452 tablet_service.cc:1514] Processing DeleteTablet for tablet 931f8edfde3d4176845798f9085a9b63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:59444
I20260809 16:39:51.196543 5780 tablet_replica.cc:330] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: stopping tablet replica
I20260809 16:39:51.196597 5780 raft_consensus.cc:2229] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.196637 5780 raft_consensus.cc:2258] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.196666 5779 ts_tablet_manager.cc:1905] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.196743 5781 ts_tablet_manager.cc:1905] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.196947 5780 ts_tablet_manager.cc:1905] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.198487 5779 ts_tablet_manager.cc:1918] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:51.198549 5779 log.cc:1196] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/931f8edfde3d4176845798f9085a9b63
I20260809 16:39:51.198678 5780 ts_tablet_manager.cc:1918] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:51.198757 5780 log.cc:1196] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/931f8edfde3d4176845798f9085a9b63
I20260809 16:39:51.198783 5779 ts_tablet_manager.cc:1939] T 931f8edfde3d4176845798f9085a9b63 P 07f7382d83624398a3b1559203ce6381: Deleting consensus metadata
I20260809 16:39:51.199000 5780 ts_tablet_manager.cc:1939] T 931f8edfde3d4176845798f9085a9b63 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting consensus metadata
I20260809 16:39:51.199080 5134 catalog_manager.cc:4841] TS 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963): tablet 931f8edfde3d4176845798f9085a9b63 (table test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]) successfully deleted
I20260809 16:39:51.199359 5781 ts_tablet_manager.cc:1918] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:51.199391 5134 catalog_manager.cc:4841] TS 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327): tablet 931f8edfde3d4176845798f9085a9b63 (table test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]) successfully deleted
I20260809 16:39:51.199412 5781 log.cc:1196] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/931f8edfde3d4176845798f9085a9b63
I20260809 16:39:51.199636 5781 ts_tablet_manager.cc:1939] T 931f8edfde3d4176845798f9085a9b63 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting consensus metadata
I20260809 16:39:51.199944 5137 catalog_manager.cc:4841] TS 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913): tablet 931f8edfde3d4176845798f9085a9b63 (table test-table [id=234ab89f60d54efb835c8fbfdbf8a6c8]) successfully deleted
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
W20260809 16:39:51.224833 5212 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:51.218204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.28.62.190:45217)
0809 16:39:51.218285 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.218287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.218461 (+ 174us) server_negotiation.cc:407] Connection header received
0809 16:39:51.218537 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.218538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.218567 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.218643 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:51.219483 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.220020 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.221141 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.221393 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.221451 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.224761 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43426: BlockingRecv error: recv got EOF from 127.0.0.1:43426 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM
I20260809 16:39:51.233731 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.225695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.28.62.190:45217)
0809 16:39:51.225737 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.225739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.226000 (+ 261us) server_negotiation.cc:407] Connection header received
0809 16:39:51.226045 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.226046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.226056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.226095 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:51.226687 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.227239 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.228199 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.228388 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.228455 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.230989 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:51.230991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:51.230991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:51.230999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:51.232560 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:51.232930 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:51.232931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:51.232931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:51.232991 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:51.233199 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:51.233200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:51.233200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:51.233315 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:51.233420 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.233536 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.233618 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:39:51.235477 5148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:51.235656 5148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:51.237067 5452 tablet_service.cc:1467] Processing CreateTablet for tablet d72d12ecc4424509a75dc1d465db5434 (DEFAULT_TABLE table=test-table [id=9f92990987f945339d076bd1272006f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.237068 5308 tablet_service.cc:1467] Processing CreateTablet for tablet d72d12ecc4424509a75dc1d465db5434 (DEFAULT_TABLE table=test-table [id=9f92990987f945339d076bd1272006f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.237123 5640 tablet_service.cc:1467] Processing CreateTablet for tablet d72d12ecc4424509a75dc1d465db5434 (DEFAULT_TABLE table=test-table [id=9f92990987f945339d076bd1272006f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.237219 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72d12ecc4424509a75dc1d465db5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.237219 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72d12ecc4424509a75dc1d465db5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.237234 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72d12ecc4424509a75dc1d465db5434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.238603 5755 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Bootstrap starting.
I20260809 16:39:51.238617 5754 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Bootstrap starting.
I20260809 16:39:51.238929 5755 tablet_bootstrap.cc:654] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.239006 5754 tablet_bootstrap.cc:654] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.239435 5758 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Bootstrap starting.
I20260809 16:39:51.239614 5755 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: No bootstrap required, opened a new log
I20260809 16:39:51.239667 5755 ts_tablet_manager.cc:1397] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:51.239776 5755 raft_consensus.cc:348] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.239823 5758 tablet_bootstrap.cc:654] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.239848 5755 raft_consensus.cc:374] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.239869 5755 raft_consensus.cc:729] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.239912 5755 consensus_queue.cc:260] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.239992 5755 ts_tablet_manager.cc:1428] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:51.240489 5758 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: No bootstrap required, opened a new log
I20260809 16:39:51.240552 5758 ts_tablet_manager.cc:1397] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:51.240703 5758 raft_consensus.cc:348] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.240794 5758 raft_consensus.cc:374] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.240823 5758 raft_consensus.cc:729] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.240868 5758 consensus_queue.cc:260] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.240963 5758 ts_tablet_manager.cc:1428] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.241490 5754 tablet_bootstrap.cc:492] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: No bootstrap required, opened a new log
I20260809 16:39:51.241562 5754 ts_tablet_manager.cc:1397] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:51.241722 5754 raft_consensus.cc:348] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.241803 5754 raft_consensus.cc:374] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.241828 5754 raft_consensus.cc:729] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.241880 5754 consensus_queue.cc:260] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.241989 5754 ts_tablet_manager.cc:1428] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.386564 5774 raft_consensus.cc:482] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:51.386674 5774 raft_consensus.cc:504] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.386861 5774 leader_election.cc:288] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.387130 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72d12ecc4424509a75dc1d465db5434" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe" is_pre_election: true
I20260809 16:39:51.387169 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72d12ecc4424509a75dc1d465db5434" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381" is_pre_election: true
I20260809 16:39:51.387283 5502 raft_consensus.cc:2454] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:51.387266 5663 raft_consensus.cc:2454] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 0.
I20260809 16:39:51.387452 5254 leader_election.cc:302] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7382d83624398a3b1559203ce6381, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.387570 5774 raft_consensus.cc:2781] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:51.387627 5774 raft_consensus.cc:482] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:51.387661 5774 raft_consensus.cc:3037] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.388382 5774 raft_consensus.cc:504] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.388521 5774 leader_election.cc:288] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Requested vote from peers 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327), 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963)
I20260809 16:39:51.388677 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72d12ecc4424509a75dc1d465db5434" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe"
I20260809 16:39:51.388715 5663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72d12ecc4424509a75dc1d465db5434" candidate_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7382d83624398a3b1559203ce6381"
I20260809 16:39:51.388783 5502 raft_consensus.cc:3037] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.388813 5663 raft_consensus.cc:3037] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.389519 5663 raft_consensus.cc:2454] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:51.389623 5502 raft_consensus.cc:2454] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8c5ca19e34a3f8d3fbe004070279a in term 1.
I20260809 16:39:51.389696 5254 leader_election.cc:302] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7382d83624398a3b1559203ce6381, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.389806 5774 raft_consensus.cc:2781] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:51.389890 5774 raft_consensus.cc:686] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Becoming Leader. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Running, Role: LEADER
I20260809 16:39:51.389947 5774 consensus_queue.cc:237] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.390581 5148 catalog_manager.cc:5495] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129). New cstate: current_term: 1 leader_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } health_report { overall_health: UNKNOWN } } }
I20260809 16:39:51.426738 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.424188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.28.62.129:43913)
0809 16:39:51.424254 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.424256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.424269 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:39:51.424371 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.424372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.424384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.424438 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.424546 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.425040 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.425974 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.426161 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.426242 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.426437 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.426470 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.426529 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.426641 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.426665 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:39:51.427665 5663 raft_consensus.cc:1261] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.427721 5502 raft_consensus.cc:1261] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Refusing update from remote peer 8ba8c5ca19e34a3f8d3fbe004070279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.427904 5766 consensus_queue.cc:1035] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.428061 5774 consensus_queue.cc:1035] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.430706 5148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.430809 5148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.431413 5148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Sending DeleteTablet for 3 replicas of tablet d72d12ecc4424509a75dc1d465db5434
I20260809 16:39:51.431670 5452 tablet_service.cc:1514] Processing DeleteTablet for tablet d72d12ecc4424509a75dc1d465db5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:59444
I20260809 16:39:51.431731 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet d72d12ecc4424509a75dc1d465db5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:39546
I20260809 16:39:51.431726 5640 tablet_service.cc:1514] Processing DeleteTablet for tablet d72d12ecc4424509a75dc1d465db5434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260809 16:39:51.431821 5779 tablet_replica.cc:330] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: stopping tablet replica
I20260809 16:39:51.431821 5781 tablet_replica.cc:330] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: stopping tablet replica
I20260809 16:39:51.431823 5780 tablet_replica.cc:330] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: stopping tablet replica
I20260809 16:39:51.431874 5779 raft_consensus.cc:2229] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.431874 5780 raft_consensus.cc:2229] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.431877 5781 raft_consensus.cc:2229] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:51.431933 5780 raft_consensus.cc:2258] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.431933 5781 raft_consensus.cc:2258] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.431951 5779 raft_consensus.cc:2258] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.432236 5779 ts_tablet_manager.cc:1905] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.432355 5781 ts_tablet_manager.cc:1905] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.432444 5148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:39:51.432567 5148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:39:51.433688 5308 tablet_service.cc:1467] Processing CreateTablet for tablet f7e66a83247141a09607e5a4077c4e42 (DEFAULT_TABLE table=test-table [id=980ef823484c4218bcfc3bc91dd0adb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.433789 5640 tablet_service.cc:1467] Processing CreateTablet for tablet f7e66a83247141a09607e5a4077c4e42 (DEFAULT_TABLE table=test-table [id=980ef823484c4218bcfc3bc91dd0adb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.433910 5780 ts_tablet_manager.cc:1905] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.433957 5781 ts_tablet_manager.cc:1918] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:51.434007 5781 log.cc:1196] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/d72d12ecc4424509a75dc1d465db5434
I20260809 16:39:51.434231 5781 ts_tablet_manager.cc:1939] T d72d12ecc4424509a75dc1d465db5434 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting consensus metadata
I20260809 16:39:51.434387 5452 tablet_service.cc:1467] Processing CreateTablet for tablet f7e66a83247141a09607e5a4077c4e42 (DEFAULT_TABLE table=test-table [id=980ef823484c4218bcfc3bc91dd0adb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:39:51.434489 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e66a83247141a09607e5a4077c4e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.435500 5780 ts_tablet_manager.cc:1918] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:51.435567 5780 log.cc:1196] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/d72d12ecc4424509a75dc1d465db5434
I20260809 16:39:51.435817 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e66a83247141a09607e5a4077c4e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.435786 5758 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Bootstrap starting.
I20260809 16:39:51.435886 5779 ts_tablet_manager.cc:1918] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:39:51.435784 5780 ts_tablet_manager.cc:1939] T d72d12ecc4424509a75dc1d465db5434 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting consensus metadata
I20260809 16:39:51.435950 5779 log.cc:1196] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/d72d12ecc4424509a75dc1d465db5434
I20260809 16:39:51.436161 5758 tablet_bootstrap.cc:654] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.436178 5779 ts_tablet_manager.cc:1939] T d72d12ecc4424509a75dc1d465db5434 P 07f7382d83624398a3b1559203ce6381: Deleting consensus metadata
I20260809 16:39:51.436453 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e66a83247141a09607e5a4077c4e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:51.437018 5755 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: Bootstrap starting.
I20260809 16:39:51.437397 5755 tablet_bootstrap.cc:654] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.437683 5754 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Bootstrap starting.
I20260809 16:39:51.437701 5134 catalog_manager.cc:4841] TS 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327): tablet d72d12ecc4424509a75dc1d465db5434 (table test-table [id=9f92990987f945339d076bd1272006f9]) successfully deleted
I20260809 16:39:51.437927 5137 catalog_manager.cc:4841] TS 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913): tablet d72d12ecc4424509a75dc1d465db5434 (table test-table [id=9f92990987f945339d076bd1272006f9]) successfully deleted
I20260809 16:39:51.438094 5755 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: No bootstrap required, opened a new log
I20260809 16:39:51.438164 5755 ts_tablet_manager.cc:1397] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:51.438273 5755 raft_consensus.cc:348] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.438333 5754 tablet_bootstrap.cc:654] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:51.438357 5755 raft_consensus.cc:374] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.438381 5755 raft_consensus.cc:729] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8c5ca19e34a3f8d3fbe004070279a, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.438431 5755 consensus_queue.cc:260] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.438530 5755 ts_tablet_manager.cc:1428] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.438748 5134 catalog_manager.cc:4841] TS 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963): tablet d72d12ecc4424509a75dc1d465db5434 (table test-table [id=9f92990987f945339d076bd1272006f9]) successfully deleted
I20260809 16:39:51.438946 5758 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: No bootstrap required, opened a new log
I20260809 16:39:51.439004 5758 ts_tablet_manager.cc:1397] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 16:39:51.439163 5758 raft_consensus.cc:348] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.439248 5758 raft_consensus.cc:374] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.439270 5758 raft_consensus.cc:729] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe5118aff94b6780e97e5220fe5dbe, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.439314 5758 consensus_queue.cc:260] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.439430 5758 ts_tablet_manager.cc:1428] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:51.439742 5754 tablet_bootstrap.cc:492] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: No bootstrap required, opened a new log
I20260809 16:39:51.439795 5754 ts_tablet_manager.cc:1397] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:39:51.439934 5754 raft_consensus.cc:348] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.440011 5754 raft_consensus.cc:374] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:51.440025 5754 raft_consensus.cc:729] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Initialized, Role: FOLLOWER
I20260809 16:39:51.440093 5754 consensus_queue.cc:260] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.440189 5754 ts_tablet_manager.cc:1428] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:51.708408 5759 raft_consensus.cc:482] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:39:51.708518 5759 raft_consensus.cc:504] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.708864 5759 leader_election.cc:288] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913), 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327)
I20260809 16:39:51.711985 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.709075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43913 (local address 127.28.62.131:50301)
0809 16:39:51.709336 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.709349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.709419 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.709586 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.709587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.709589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:51.709675 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.709676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.710305 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.710306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.711152 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.711154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.711692 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.711784 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.711789 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.711841 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.711870 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260809 16:39:51.711992 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.709125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:50301 (local address 127.28.62.129:43913)
0809 16:39:51.709175 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.709177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.709407 (+ 230us) server_negotiation.cc:407] Connection header received
0809 16:39:51.709507 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.709509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.709520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.709575 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:51.709750 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.710196 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711240 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.711612 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711668 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.711675 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.711867 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.711918 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:39:51.712085 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.709141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:60153 (local address 127.28.62.130:44327)
0809 16:39:51.709347 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.709350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.709360 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:51.709395 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.709396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.709409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.709492 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:39:51.709669 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.710187 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711290 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.711687 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711762 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.711772 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.711933 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.711987 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260809 16:39:51.712324 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.709068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.28.62.131:60153)
0809 16:39:51.709215 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.709229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.709298 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.709514 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.709515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.709518 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:39:51.709586 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.709588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.710277 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.710279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.711174 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.711177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.711779 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.711780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.711845 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.711851 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.711909 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.711944 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260809 16:39:51.712380 5333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e66a83247141a09607e5a4077c4e42" candidate_uuid: "07f7382d83624398a3b1559203ce6381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" is_pre_election: true
I20260809 16:39:51.712510 5333 raft_consensus.cc:2454] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f7382d83624398a3b1559203ce6381 in term 0.
I20260809 16:39:51.712617 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e66a83247141a09607e5a4077c4e42" candidate_uuid: "07f7382d83624398a3b1559203ce6381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe" is_pre_election: true
I20260809 16:39:51.712706 5502 raft_consensus.cc:2454] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f7382d83624398a3b1559203ce6381 in term 0.
I20260809 16:39:51.712721 5584 leader_election.cc:302] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7382d83624398a3b1559203ce6381, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.712865 5759 raft_consensus.cc:2781] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:39:51.712910 5759 raft_consensus.cc:482] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:39:51.712953 5759 raft_consensus.cc:3037] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.713660 5759 raft_consensus.cc:504] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.713825 5759 leader_election.cc:288] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [CANDIDATE]: Term 1 election: Requested vote from peers 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913), 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327)
I20260809 16:39:51.713979 5333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e66a83247141a09607e5a4077c4e42" candidate_uuid: "07f7382d83624398a3b1559203ce6381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a"
I20260809 16:39:51.713980 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e66a83247141a09607e5a4077c4e42" candidate_uuid: "07f7382d83624398a3b1559203ce6381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe5118aff94b6780e97e5220fe5dbe"
I20260809 16:39:51.714059 5502 raft_consensus.cc:3037] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.714058 5333 raft_consensus.cc:3037] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:51.714835 5502 raft_consensus.cc:2454] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f7382d83624398a3b1559203ce6381 in term 1.
I20260809 16:39:51.714835 5333 raft_consensus.cc:2454] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f7382d83624398a3b1559203ce6381 in term 1.
I20260809 16:39:51.715001 5584 leader_election.cc:302] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7382d83624398a3b1559203ce6381, 8ba8c5ca19e34a3f8d3fbe004070279a; no voters:
I20260809 16:39:51.715126 5759 raft_consensus.cc:2781] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:51.715260 5759 raft_consensus.cc:686] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 1 LEADER]: Becoming Leader. State: Replica: 07f7382d83624398a3b1559203ce6381, State: Running, Role: LEADER
I20260809 16:39:51.715329 5759 consensus_queue.cc:237] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } }
I20260809 16:39:51.716161 5148 catalog_manager.cc:5495] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07f7382d83624398a3b1559203ce6381 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "07f7382d83624398a3b1559203ce6381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07f7382d83624398a3b1559203ce6381" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 41963 } health_report { overall_health: HEALTHY } } }
I20260809 16:39:51.723251 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.720658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.28.62.131:41963)
0809 16:39:51.720842 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.720844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.720853 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:51.720891 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.720892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.720920 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.720993 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.721112 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.721664 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.722498 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.722747 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.722800 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.722891 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.722992 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.723119 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.723151 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.723167 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260809 16:39:51.728070 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.725595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41599 (local address 127.28.62.130:44327)
0809 16:39:51.725651 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.725654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.725700 (+ 46us) server_negotiation.cc:407] Connection header received
0809 16:39:51.725773 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.725774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.725785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.725834 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.725999 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.726463 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.727308 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.727505 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.727595 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.727800 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.727837 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.727899 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.727960 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.727983 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:39:51.728127 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.725475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44327 (local address 127.28.62.131:41599)
0809 16:39:51.725636 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.725649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.725711 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.725849 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.725849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.725851 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:51.725916 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.725917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.726553 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.726555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.727193 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.727195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.727618 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.727619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.727683 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.727697 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.727899 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.727902 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.727956 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.728031 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260809 16:39:51.730366 5502 raft_consensus.cc:1261] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Refusing update from remote peer 07f7382d83624398a3b1559203ce6381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.730400 5333 raft_consensus.cc:1261] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Refusing update from remote peer 07f7382d83624398a3b1559203ce6381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:39:51.730619 5850 consensus_queue.cc:1035] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fe5118aff94b6780e97e5220fe5dbe" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.730748 5759 consensus_queue.cc:1035] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba8c5ca19e34a3f8d3fbe004070279a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:39:51.738459 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.736208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:41563 (local address 127.28.62.131:41963)
0809 16:39:51.736258 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:51.736265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:51.736353 (+ 88us) server_negotiation.cc:407] Connection header received
0809 16:39:51.736407 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:51.736407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:51.736417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:51.736465 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:39:51.736548 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.737011 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.737790 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.738081 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.738149 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.738274 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.738307 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.738355 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:51.738386 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 16:39:51.738406 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:39:51.738479 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.736171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:41963 (local address 127.28.62.130:41563)
0809 16:39:51.736296 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.736307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:51.736367 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:51.736468 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:51.736469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:51.736470 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 16:39:51.736510 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.736511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.737079 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.737080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.737652 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:51.737654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:51.738155 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:51.738155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:51.738211 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:51.738215 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 16:39:51.738354 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 16:39:51.738355 (+ 1us) client_negotiation.cc:769] Sending connection context
0809 16:39:51.738383 (+ 28us) client_negotiation.cc:240] Negotiation successful
0809 16:39:51.738420 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260809 16:39:51.745146 5150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.745273 5150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:39:51.745934 5150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320833cea4e844078996cf859117e170: Sending DeleteTablet for 3 replicas of tablet f7e66a83247141a09607e5a4077c4e42
I20260809 16:39:51.746215 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e66a83247141a09607e5a4077c4e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:39546
I20260809 16:39:51.746209 5640 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e66a83247141a09607e5a4077c4e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260809 16:39:51.746333 5779 tablet_replica.cc:330] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: stopping tablet replica
I20260809 16:39:51.746333 5781 tablet_replica.cc:330] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: stopping tablet replica
I20260809 16:39:51.746382 5779 raft_consensus.cc:2229] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:39:51.746387 5781 raft_consensus.cc:2229] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.746426 5781 raft_consensus.cc:2258] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.746470 5779 raft_consensus.cc:2258] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.746791 5779 ts_tablet_manager.cc:1905] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.746891 5781 ts_tablet_manager.cc:1905] T f7e66a83247141a09607e5a4077c4e42 P 8ba8c5ca19e34a3f8d3fbe004070279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.747224 5452 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e66a83247141a09607e5a4077c4e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:39:51 UTC) from {username='oryx', principal='oryx/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:59444
I20260809 16:39:51.747331 5780 tablet_replica.cc:330] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: stopping tablet replica
I20260809 16:39:51.747380 5780 raft_consensus.cc:2229] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:39:51.747416 5780 raft_consensus.cc:2258] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:39:51.748431 5779 ts_tablet_manager.cc:1918] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:51.748493 5779 log.cc:1196] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/f7e66a83247141a09607e5a4077c4e42
I20260809 16:39:51.748641 5780 ts_tablet_manager.cc:1905] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:39:51.748733 5779 ts_tablet_manager.cc:1939] T f7e66a83247141a09607e5a4077c4e42 P 07f7382d83624398a3b1559203ce6381: Deleting consensus metadata
I20260809 16:39:51.748970 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 5236
I20260809 16:39:51.749051 5134 catalog_manager.cc:4841] TS 07f7382d83624398a3b1559203ce6381 (127.28.62.131:41963): tablet f7e66a83247141a09607e5a4077c4e42 (table test-table [id=980ef823484c4218bcfc3bc91dd0adb7]) successfully deleted
I20260809 16:39:51.753505 5780 ts_tablet_manager.cc:1918] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 16:39:51.753594 5780 log.cc:1196] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/f7e66a83247141a09607e5a4077c4e42
I20260809 16:39:51.753887 5780 ts_tablet_manager.cc:1939] T f7e66a83247141a09607e5a4077c4e42 P 57fe5118aff94b6780e97e5220fe5dbe: Deleting consensus metadata
I20260809 16:39:51.754310 5134 catalog_manager.cc:4841] TS 57fe5118aff94b6780e97e5220fe5dbe (127.28.62.130:44327): tablet f7e66a83247141a09607e5a4077c4e42 (table test-table [id=980ef823484c4218bcfc3bc91dd0adb7]) successfully deleted
W20260809 16:39:51.759704 5137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:43913 (error 108)
I20260809 16:39:51.759992 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 5404
I20260809 16:39:51.761253 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:51.760141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43913 (local address 127.0.0.1:39580)
0809 16:39:51.761159 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:51.761216 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:43913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":41,"threads_started":1}
W20260809 16:39:51.761370 5137 catalog_manager.cc:4584] TS 8ba8c5ca19e34a3f8d3fbe004070279a (127.28.62.129:43913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7e66a83247141a09607e5a4077c4e42: Network error: Client connection negotiation failed: client connection to 127.28.62.129:43913: connect: Connection refused (error 111)
I20260809 16:39:51.768714 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 5561
I20260809 16:39:51.777040 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 5116
2026-08-09T16:39:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3207 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5872](info): set up 2 sockets
Aug 09 16:39:51 dist-test-slave-wl2h krb5kdc[5872](info): commencing operation
krb5kdc: starting...
W20260809 16:39:52.945817 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:53 dist-test-slave-wl2h krb5kdc[5872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:53.078229 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35533
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:53.161695 5887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:53.161968 5887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:53.162007 5887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:53.163228 5887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:53.163298 5887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:53.163323 5887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:53.163349 5887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:53.163381 5887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:53.164892 5887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:53.165185 5887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:53.165439 5887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:53.170650 5896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:53.170678 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:53.170742 5887 server_base.cc:1034] running on GCE node
W20260809 16:39:53.170677 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:53.171249 5887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:53.171571 5887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:53.172788 5887 hybrid_clock.cc:648] HybridClock initialized: now 1786293593172785 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T16:39:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1397 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:39:53 dist-test-slave-wl2h krb5kdc[5901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:39:53 dist-test-slave-wl2h krb5kdc[5901](info): set up 2 sockets
Aug 09 16:39:53 dist-test-slave-wl2h krb5kdc[5901](info): commencing operation
krb5kdc: starting...
W20260809 16:39:54.396260 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:39:54.522629 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44015
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:54.606247 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:54.606453 5918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:54.606475 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:54.607638 5918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:54.607722 5918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:54.607752 5918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:54.607774 5918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:54.607805 5918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:54.609328 5918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:54.609586 5918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:54.609830 5918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:54.614825 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:54.614836 5925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:54.614823 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:54.615301 5918 server_base.cc:1034] running on GCE node
I20260809 16:39:54.615545 5918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:54.615837 5918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:54.617020 5918 hybrid_clock.cc:648] HybridClock initialized: now 1786293594617011 us; error 41 us; skew 500 ppm
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:54.619717 5918 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:54.620157 5918 webserver.cc:460] Webserver started at http://127.28.62.190:34229/ using document root <none> and password file <none>
I20260809 16:39:54.620409 5918 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:54.620481 5918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:54.620657 5918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:54.621729 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "69e283d4dc0e445bbd6e232efc66c789"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "e9b089636cffe2eb111554322111d09f"
server_key_iv: "7ac0013e362e114113c8c45fdad8b297"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.622102 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "69e283d4dc0e445bbd6e232efc66c789"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "e9b089636cffe2eb111554322111d09f"
server_key_iv: "7ac0013e362e114113c8c45fdad8b297"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.623555 5918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:54.624357 5935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:54.624558 5918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:54.624670 5918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "69e283d4dc0e445bbd6e232efc66c789"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "e9b089636cffe2eb111554322111d09f"
server_key_iv: "7ac0013e362e114113c8c45fdad8b297"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.624747 5918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:54.639909 5918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:54.641964 5918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:54.642143 5918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:54.646528 5918 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44015
I20260809 16:39:54.646570 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44015 every 8 connection(s)
I20260809 16:39:54.646948 5918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:54.647943 5998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:39:54.648514 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 5918
I20260809 16:39:54.648676 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:39:54.648880 28922 external_mini_cluster.cc:1428] Setting key c39aa34946d5c8c13b3f7e180b3bfab5
I20260809 16:39:54.650736 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Bootstrap starting.
I20260809 16:39:54.651386 5998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Neither blocks nor log segments found. Creating new log.
I20260809 16:39:54.651706 5998 log.cc:824] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:54.652419 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: No bootstrap required, opened a new log
I20260809 16:39:54.654428 5998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:54.654590 5998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:39:54.654645 5998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e283d4dc0e445bbd6e232efc66c789, State: Initialized, Role: FOLLOWER
I20260809 16:39:54.654800 5998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:54.654898 5998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:54.654943 5998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:54.654991 5998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:39:54.655743 5998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:54.655857 5998 leader_election.cc:302] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e283d4dc0e445bbd6e232efc66c789; no voters:
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:54.656032 5998 leader_election.cc:288] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:39:54.656066 6005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:39:54.656189 6005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 LEADER]: Becoming Leader. State: Replica: 69e283d4dc0e445bbd6e232efc66c789, State: Running, Role: LEADER
I20260809 16:39:54.656323 5998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:54.656307 6005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:54.656672 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e283d4dc0e445bbd6e232efc66c789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } } }
I20260809 16:39:54.657152 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:54.657313 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e283d4dc0e445bbd6e232efc66c789. Latest consensus state: current_term: 1 leader_uuid: "69e283d4dc0e445bbd6e232efc66c789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } } }
I20260809 16:39:54.657471 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:54.657390 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:54.658414 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:54.660068 6015 catalog_manager.cc:1347] Generated new cluster ID: eb610ff3c96049998737378813b5975d
I20260809 16:39:54.660120 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:54.660776 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:54.649459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.28.62.190:44015)
0809 16:39:54.649663 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:54.649671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:54.649695 (+ 24us) server_negotiation.cc:407] Connection header received
0809 16:39:54.650113 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:54.650118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:54.650163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:54.650314 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:54.650998 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.652086 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.652928 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.653355 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.653483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:54.656512 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:54.656518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:54.656519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:54.656535 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:54.658718 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:54.659163 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:54.659165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:54.659166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:54.659266 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:54.659506 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:54.659508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:54.659509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:54.659819 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:54.659962 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:54.660423 (+ 461us) server_negotiation.cc:299] Negotiation successful
0809 16:39:54.660581 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260809 16:39:54.685204 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:39:54.685984 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:54.693019 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Generated new TSK 0
I20260809 16:39:54.693229 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:39:54.729614 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:54.818192 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:54.818447 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:54.818490 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:54.819710 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:54.819779 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:54.819829 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:54.821363 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:54.821636 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:54.821877 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:54.827241 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:54.827268 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:54.827265 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:54.827598 6033 server_base.cc:1034] running on GCE node
I20260809 16:39:54.827854 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:54.828109 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:54.829278 6033 hybrid_clock.cc:648] HybridClock initialized: now 1786293594829265 us; error 30 us; skew 500 ppm
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:54.832000 6033 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:39:54.832479 6033 webserver.cc:460] Webserver started at http://127.28.62.129:41533/ using document root <none> and password file <none>
I20260809 16:39:54.832726 6033 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:54.832798 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:54.832952 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:54.834013 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "1f56fe0de7a044a885c36cdfc1db1da6"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "eebe6e5c882465fbb349d51765def96e"
server_key_iv: "7746a4f8b0c55dd1a00c9208d73ddb03"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.834385 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "1f56fe0de7a044a885c36cdfc1db1da6"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "eebe6e5c882465fbb349d51765def96e"
server_key_iv: "7746a4f8b0c55dd1a00c9208d73ddb03"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.835896 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:39:54.836717 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:54.836900 6033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:54.837007 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "1f56fe0de7a044a885c36cdfc1db1da6"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "eebe6e5c882465fbb349d51765def96e"
server_key_iv: "7746a4f8b0c55dd1a00c9208d73ddb03"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.837090 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:54.845911 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:54.847976 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:54.848146 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:54.848392 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:54.848722 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:54.848775 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:54.848811 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:54.848837 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:54.854885 6033 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:40145
I20260809 16:39:54.854964 6185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:40145 every 8 connection(s)
I20260809 16:39:54.855315 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:39:54.855937 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6033
I20260809 16:39:54.856070 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:39:54.856437 28922 external_mini_cluster.cc:1428] Setting key c4944476a20e4fd19963ff3d4ff4d344
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:39:54.867082 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:54.856837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:54927 (local address 127.28.62.190:44015)
0809 16:39:54.856922 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:54.856924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:54.857385 (+ 461us) server_negotiation.cc:407] Connection header received
0809 16:39:54.857913 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:54.857914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:54.857928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:54.857998 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:54.858976 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.859644 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.861163 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.861377 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.861436 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:54.864039 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:54.864040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:54.864041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:54.864049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:54.865496 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:54.865927 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:54.865928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:54.865929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:54.865992 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:54.866315 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:54.866316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:54.866316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:54.866436 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:54.866522 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:54.866827 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 16:39:54.866923 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 16:39:54.867229 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:54.856824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:54927)
0809 16:39:54.857054 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:54.857290 (+ 236us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:54.857816 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:54.858086 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:54.858089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:54.858682 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:54.858878 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:54.858882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.859744 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.859745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:54.861053 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:54.861056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:54.861470 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:54.861472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:54.861655 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:54.862240 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:54.862249 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:54.863934 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:54.865588 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:54.865596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:54.865598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:54.865843 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:54.866071 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:54.866072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:54.866072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:54.866187 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:54.866511 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:54.866519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:54.866670 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 16:39:54.866800 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 16:39:54.866965 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":137,"threads_started":1}
I20260809 16:39:54.868211 6186 heartbeater.cc:347] Connected to a master server at 127.28.62.190:44015
I20260809 16:39:54.868350 6186 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:54.868583 6186 heartbeater.cc:511] Master 127.28.62.190:44015 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
I20260809 16:39:54.869567 5954 ts_manager.cc:194] Registered new tserver with Master: 1f56fe0de7a044a885c36cdfc1db1da6 (127.28.62.129:40145)
I20260809 16:39:54.870242 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:54927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:39:54.890224 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:54.974201 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:54.974474 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:54.974514 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:54.975706 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:54.975782 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:54.975829 6191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:54.977324 6191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:54.977617 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:54.977872 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:54.982914 6196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:54.982950 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:54.983284 6191 server_base.cc:1034] running on GCE node
W20260809 16:39:54.982950 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:54.983538 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:54.983767 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:54.984928 6191 hybrid_clock.cc:648] HybridClock initialized: now 1786293594984915 us; error 32 us; skew 500 ppm
Aug 09 16:39:54 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:54.988214 6191 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:39:54.988732 6191 webserver.cc:460] Webserver started at http://127.28.62.130:38839/ using document root <none> and password file <none>
I20260809 16:39:54.988997 6191 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:54.989073 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:54.989225 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:54.990360 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "b461252f568244aeb71dd56f7ebd326a"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "2b4518be599f99261c9ce993100bf7ec"
server_key_iv: "9257b89f9601d8686b921834db469b5c"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.990741 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "b461252f568244aeb71dd56f7ebd326a"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "2b4518be599f99261c9ce993100bf7ec"
server_key_iv: "9257b89f9601d8686b921834db469b5c"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.992251 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:39:54.993148 6207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:54.993376 6191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:54.993484 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b461252f568244aeb71dd56f7ebd326a"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "2b4518be599f99261c9ce993100bf7ec"
server_key_iv: "9257b89f9601d8686b921834db469b5c"
server_key_version: "encryptionkey@0"
I20260809 16:39:54.993562 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.003304 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.005354 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.005533 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.005798 6191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:55.006129 6191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:55.006186 6191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.006222 6191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:55.006269 6191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.012923 6191 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:41203
I20260809 16:39:55.012967 6341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:41203 every 8 connection(s)
I20260809 16:39:55.013305 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:55.016616 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6191
I20260809 16:39:55.016728 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:39:55.016933 28922 external_mini_cluster.cc:1428] Setting key 016f329473b5b30c36b6c3b93a21ddc6
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293594, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:55.024971 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.014101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:49735 (local address 127.28.62.190:44015)
0809 16:39:55.014169 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.014172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.014585 (+ 413us) server_negotiation.cc:407] Connection header received
0809 16:39:55.015178 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.015179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.015191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.015267 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.016315 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.016907 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.018291 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.018479 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.018535 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.021505 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:55.021507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:55.021507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:55.021514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:55.023153 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.023748 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.023749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.023750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.023820 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.024204 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.024205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.024206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.024330 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:55.024399 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:55.024818 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 16:39:55.024903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 16:39:55.025295 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.014085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:49735)
0809 16:39:55.014413 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.014465 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.015003 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.015330 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.015333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.015979 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.016201 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.016205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.017013 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.017015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.018206 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.018209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.018568 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.018570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.018757 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.019431 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.019443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.021403 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:55.023239 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:55.023247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:55.023250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:55.023628 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:55.023906 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:55.023907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:55.023907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:55.024090 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:55.024418 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:55.024427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:55.024631 (+ 204us) client_negotiation.cc:769] Sending connection context
0809 16:39:55.024784 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 16:39:55.024995 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":243,"threads_started":1}
I20260809 16:39:55.026408 6342 heartbeater.cc:347] Connected to a master server at 127.28.62.190:44015
I20260809 16:39:55.026510 6342 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:55.026713 6342 heartbeater.cc:511] Master 127.28.62.190:44015 requested a full tablet report, sending...
I20260809 16:39:55.027150 5954 ts_manager.cc:194] Registered new tserver with Master: b461252f568244aeb71dd56f7ebd326a (127.28.62.130:41203)
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:39:55.027743 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:49735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:39:55.047318 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:55.133711 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:55.133906 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:55.133929 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:55.135083 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:55.135125 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:55.135151 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:55.136526 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:55.136708 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:55.136893 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:55.142319 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.142482 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.142478 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:55.142647 6351 server_base.cc:1034] running on GCE node
I20260809 16:39:55.142838 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:55.143102 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:55.144244 6351 hybrid_clock.cc:648] HybridClock initialized: now 1786293595144230 us; error 30 us; skew 500 ppm
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:55.146711 6351 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:39:55.147229 6351 webserver.cc:460] Webserver started at http://127.28.62.131:42731/ using document root <none> and password file <none>
I20260809 16:39:55.147449 6351 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:55.147490 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:55.147579 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:39:55.148589 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "f01b369da4fd4cda8e79ea75fe1f7ad2"
format_stamp: "Formatted at 2026-08-09 16:39:55 on dist-test-slave-wl2h"
server_key: "4ba62448fb22d067f0c98700328ecc4c"
server_key_iv: "b6fb5c8c752130d0517b263e39f59f73"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.148898 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "f01b369da4fd4cda8e79ea75fe1f7ad2"
format_stamp: "Formatted at 2026-08-09 16:39:55 on dist-test-slave-wl2h"
server_key: "4ba62448fb22d067f0c98700328ecc4c"
server_key_iv: "b6fb5c8c752130d0517b263e39f59f73"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.150187 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:39:55.150902 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.151099 6351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:39:55.151167 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "f01b369da4fd4cda8e79ea75fe1f7ad2"
format_stamp: "Formatted at 2026-08-09 16:39:55 on dist-test-slave-wl2h"
server_key: "4ba62448fb22d067f0c98700328ecc4c"
server_key_iv: "b6fb5c8c752130d0517b263e39f59f73"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.151261 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.177207 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.179665 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.179826 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.180104 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:55.180441 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:55.180476 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.180500 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:55.180549 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.186467 6351 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:34757
I20260809 16:39:55.186517 6499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:34757 every 8 connection(s)
I20260809 16:39:55.186834 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:39:55.192999 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6351
I20260809 16:39:55.193135 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:39:55.193341 28922 external_mini_cluster.cc:1428] Setting key 618c0e62d108fa4ddae3ad2a18a4e666
I20260809 16:39:55.196568 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.187602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:53193 (local address 127.28.62.190:44015)
0809 16:39:55.187696 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.187699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.187874 (+ 175us) server_negotiation.cc:407] Connection header received
0809 16:39:55.188418 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.188419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.188433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.188485 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.189487 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.190016 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.191039 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.191255 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.191331 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.193734 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:55.193735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:55.193735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:55.193743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:55.195261 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.195647 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.195648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.195648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.195699 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.195909 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.195909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.195910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.196012 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:55.196062 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:55.196439 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 16:39:55.196504 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 16:39:55.196827 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.187550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:53193)
0809 16:39:55.187744 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.187783 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.188303 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.188557 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.188561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.189185 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.189383 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.189388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.190114 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.190115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.190918 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.190920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.191339 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.191340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.191466 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.191950 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.191959 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.193616 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:39:55.195350 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:55.195356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:55.195358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:55.195582 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:55.195759 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:39:55.195759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:39:55.195760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:39:55.195855 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:39:55.196092 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:39:55.196097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:39:55.196256 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 16:39:55.196416 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 16:39:55.196559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260809 16:39:55.197352 6500 heartbeater.cc:347] Connected to a master server at 127.28.62.190:44015
I20260809 16:39:55.197453 6500 heartbeater.cc:464] Registering TS with master...
I20260809 16:39:55.197638 6500 heartbeater.cc:511] Master 127.28.62.190:44015 requested a full tablet report, sending...
I20260809 16:39:55.198045 5954 ts_manager.cc:194] Registered new tserver with Master: f01b369da4fd4cda8e79ea75fe1f7ad2 (127.28.62.131:34757)
I20260809 16:39:55.198392 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:53193
I20260809 16:39:55.205511 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
Entry for principal oryx/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab.
WARNING: no policy specified for oryx/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
Entry for principal oryx/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab.
WARNING: no policy specified for oryx/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
Entry for principal oryx/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab.
I20260809 16:39:55.282671 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6033
I20260809 16:39:55.296494 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6191
I20260809 16:39:55.305001 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6351
I20260809 16:39:55.315672 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 5918
I20260809 16:39:55.324985 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44015
--webserver_interface=127.28.62.190
--webserver_port=34229
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:39:55.412436 6510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:55.412678 6510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:55.412704 6510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:55.413990 6510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:39:55.414041 6510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:55.414058 6510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:55.414073 6510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:39:55.414086 6510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:39:55.415807 6510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=34229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:55.416100 6510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:55.416374 6510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:55.421685 6517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.421693 6519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.421693 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:55.421919 6510 server_base.cc:1034] running on GCE node
I20260809 16:39:55.422149 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:55.422423 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:55.423599 6510 hybrid_clock.cc:648] HybridClock initialized: now 1786293595423584 us; error 29 us; skew 500 ppm
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:55.426275 6510 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:39:55.426780 6510 webserver.cc:460] Webserver started at http://127.28.62.190:34229/ using document root <none> and password file <none>
I20260809 16:39:55.427078 6510 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:55.427171 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:55.428614 6510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:55.429328 6529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.429512 6510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:55.429632 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "69e283d4dc0e445bbd6e232efc66c789"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "e9b089636cffe2eb111554322111d09f"
server_key_iv: "7ac0013e362e114113c8c45fdad8b297"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.429996 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.450202 6510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.452791 6510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.452971 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.457530 6510 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44015
I20260809 16:39:55.457571 6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44015 every 8 connection(s)
I20260809 16:39:55.457990 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:39:55.459643 6587 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:39:55.460357 6587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Bootstrap starting.
I20260809 16:39:55.460676 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6510
I20260809 16:39:55.461014 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:40145
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:44015
--webserver_port=41533
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:39:55.469493 6587 log.cc:824] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Log is configured to *not* fsync() on all Append() calls
I20260809 16:39:55.471433 6587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:39:55.471737 6587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Bootstrap complete.
I20260809 16:39:55.474128 6587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:55.474392 6587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e283d4dc0e445bbd6e232efc66c789, State: Initialized, Role: FOLLOWER
I20260809 16:39:55.474606 6587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:55.474707 6587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:39:55.474761 6587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:39:55.474812 6587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:39:55.475894 6587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:55.476027 6587 leader_election.cc:302] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e283d4dc0e445bbd6e232efc66c789; no voters:
I20260809 16:39:55.476245 6587 leader_election.cc:288] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 16:39:55.476292 6592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 16:39:55.476411 6592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [term 2 LEADER]: Becoming Leader. State: Replica: 69e283d4dc0e445bbd6e232efc66c789, State: Running, Role: LEADER
I20260809 16:39:55.476537 6587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:39:55.476567 6592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } }
I20260809 16:39:55.476886 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e283d4dc0e445bbd6e232efc66c789. Latest consensus state: current_term: 2 leader_uuid: "69e283d4dc0e445bbd6e232efc66c789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } } }
I20260809 16:39:55.476981 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:55.476890 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69e283d4dc0e445bbd6e232efc66c789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e283d4dc0e445bbd6e232efc66c789" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44015 } } }
I20260809 16:39:55.477156 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789 [sys.catalog]: This master's current role is: LEADER
I20260809 16:39:55.477195 6598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:39:55.477731 6598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:39:55.477953 6598 catalog_manager.cc:1259] Loaded cluster ID: eb610ff3c96049998737378813b5975d
I20260809 16:39:55.478142 6598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:39:55.479331 6598 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:39:55.479506 6598 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69e283d4dc0e445bbd6e232efc66c789: Loaded TSK: 0
I20260809 16:39:55.479720 6598 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 16:39:55.547065 6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:55.547329 6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:55.547371 6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:55.548565 6589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:55.548643 6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:55.548688 6589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:39:55.550307 6589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.129.keytab
--principal=oryx/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=41533
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:55.550608 6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:55.550866 6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:55.556047 6621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.556070 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.556162 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:55.556524 6589 server_base.cc:1034] running on GCE node
I20260809 16:39:55.556766 6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:55.557027 6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:55.558192 6589 hybrid_clock.cc:648] HybridClock initialized: now 1786293595558174 us; error 34 us; skew 500 ppm
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:55.560762 6589 init.cc:377] Logged in from keytab as oryx/127.28.62.129@KRBTEST.COM (short username oryx)
I20260809 16:39:55.561264 6589 webserver.cc:460] Webserver started at http://127.28.62.129:41533/ using document root <none> and password file <none>
I20260809 16:39:55.561519 6589 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:55.561584 6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:55.562997 6589 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:39:55.563694 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.563865 6589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:55.563968 6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "1f56fe0de7a044a885c36cdfc1db1da6"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "eebe6e5c882465fbb349d51765def96e"
server_key_iv: "7746a4f8b0c55dd1a00c9208d73ddb03"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.564404 6589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.586684 6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.589634 6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.589814 6589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.590087 6589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:55.590447 6589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:55.590499 6589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.590533 6589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:55.590551 6589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.598273 6589 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:40145
I20260809 16:39:55.598307 6764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:40145 every 8 connection(s)
I20260809 16:39:55.598752 6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.607182 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6589
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.607549 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:41203
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:44015
--webserver_port=38839
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:39:55.608484 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.599548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:43579)
0809 16:39:55.599776 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.599825 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.600459 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.600880 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.600884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.601614 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.601890 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.601895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.602993 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.602994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.604241 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.604244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.604834 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.604835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.604979 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.605546 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.605555 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.608279 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":120,"threads_started":1}
W20260809 16:39:55.608843 6765 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:44015 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.62.190:44015: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
W20260809 16:39:55.609477 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.599698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:43579 (local address 127.28.62.190:44015)
0809 16:39:55.599894 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.599901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.599931 (+ 30us) server_negotiation.cc:407] Connection header received
0809 16:39:55.600641 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.600646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.600695 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.600801 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.601998 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.602885 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.604323 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.604740 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.604822 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.608455 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:43579: BlockingRecv error: recv got EOF from 127.28.62.129:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.614943 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.609092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:47615)
0809 16:39:55.609528 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.609541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.609594 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.609855 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.609856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.609990 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.610037 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.610038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.610708 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.610709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.611573 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.611576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.612093 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.612094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.612187 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.612667 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.612673 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.614870 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415}
W20260809 16:39:55.615111 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.609169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47615 (local address 127.28.62.190:44015)
0809 16:39:55.609703 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.609705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.609712 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:39:55.609738 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.609738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.609749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.609844 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.610130 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.610608 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.611680 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.611989 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.612091 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.614995 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:47615: BlockingRecv error: recv got EOF from 127.28.62.129:47615 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":116,"threads_started":1}
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.620880 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.615402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:35819)
0809 16:39:55.615450 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.615461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.615500 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.615667 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.615668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.615848 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.615883 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.615885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.616867 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.616868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.617661 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.617662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.618179 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.618180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.618235 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.618755 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.618761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.620808 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 16:39:55.621009 6765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:39:55.621142 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.615465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:35819 (local address 127.28.62.190:44015)
0809 16:39:55.615503 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.615504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.615513 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:55.615554 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.615555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.615565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.615623 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.615973 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.616777 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.617760 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.618089 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.618171 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.621084 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:35819: BlockingRecv error: recv got EOF from 127.28.62.129:35819 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260809 16:39:55.693475 6770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:55.693689 6770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:55.693746 6770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:55.694964 6770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:55.695052 6770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:55.695101 6770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:39:55.696525 6770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.130.keytab
--principal=oryx/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:41203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=38839
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:55.696801 6770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:55.697041 6770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:55.702104 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.702150 6782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.702131 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:55.702216 6770 server_base.cc:1034] running on GCE node
I20260809 16:39:55.702515 6770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:55.702730 6770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:55.703856 6770 hybrid_clock.cc:648] HybridClock initialized: now 1786293595703830 us; error 50 us; skew 500 ppm
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:55.706254 6770 init.cc:377] Logged in from keytab as oryx/127.28.62.130@KRBTEST.COM (short username oryx)
I20260809 16:39:55.706740 6770 webserver.cc:460] Webserver started at http://127.28.62.130:38839/ using document root <none> and password file <none>
I20260809 16:39:55.706974 6770 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:55.707084 6770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:55.708974 6770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:39:55.709582 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.709758 6770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:39:55.709857 6770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b461252f568244aeb71dd56f7ebd326a"
format_stamp: "Formatted at 2026-08-09 16:39:54 on dist-test-slave-wl2h"
server_key: "2b4518be599f99261c9ce993100bf7ec"
server_key_iv: "9257b89f9601d8686b921834db469b5c"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.710218 6770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.729633 6770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.731702 6770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.731865 6770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.732151 6770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:55.732488 6770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:55.732549 6770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.732592 6770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:55.732620 6770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.738961 6770 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:41203
I20260809 16:39:55.739056 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:41203 every 8 connection(s)
I20260809 16:39:55.739423 6770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:39:55.744634 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6770
I20260809 16:39:55.744939 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:34757
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:44015
--webserver_port=42731
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.747174 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.740152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:48263)
0809 16:39:55.740341 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.740379 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.740897 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.741152 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.741156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.741748 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.741936 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.741941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.742651 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.742652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.743478 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.743480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.743873 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.743874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.743998 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.744543 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.744551 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.746959 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
W20260809 16:39:55.747190 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.740191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:48263 (local address 127.28.62.190:44015)
0809 16:39:55.740263 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.740265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.740476 (+ 211us) server_negotiation.cc:407] Connection header received
0809 16:39:55.741008 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.741009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.741026 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.741085 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.742046 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.742542 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.743572 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.743764 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.743859 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.747102 (+ 3243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:48263: BlockingRecv error: recv got EOF from 127.28.62.130:48263 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260809 16:39:55.747467 6922 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:44015 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.62.190:44015: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.752394 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.747735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:47385)
0809 16:39:55.747774 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.747784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.747848 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.747965 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.747966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.748098 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.748144 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.748145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.748763 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.748764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.749313 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.749316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.749647 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.749647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.749717 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.750234 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.750239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.752270 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260809 16:39:55.752512 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.747785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:47385 (local address 127.28.62.190:44015)
0809 16:39:55.747848 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.747850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.747859 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:55.747904 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.747905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.747917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.747964 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.748227 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.748682 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.749384 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.749574 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.749649 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.752434 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:47385: BlockingRecv error: recv got EOF from 127.28.62.130:47385 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.757706 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.752797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:34965)
0809 16:39:55.752907 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.752920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.752982 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.753135 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.753136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.753290 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.753329 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.753331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.753936 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.753937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.754816 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.754819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.755227 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.755228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.755291 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.755704 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.755710 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.757621 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260809 16:39:55.757814 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.752852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:34965 (local address 127.28.62.190:44015)
0809 16:39:55.752911 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.752913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.752974 (+ 61us) server_negotiation.cc:407] Connection header received
0809 16:39:55.753050 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.753051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.753060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.753107 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.753411 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.753856 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.754908 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.755141 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.755198 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.757722 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:34965: BlockingRecv error: recv got EOF from 127.28.62.130:34965 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 16:39:55.758023 6922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:39:55.830312 6926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:39:55.830580 6926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:39:55.830621 6926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:39:55.831831 6926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:39:55.831905 6926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:39:55.831951 6926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:39:55.833431 6926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/krb5kdc/oryx_127.28.62.131.keytab
--principal=oryx/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:34757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=42731
--tserver_master_addrs=127.28.62.190:44015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:39:55.833716 6926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:39:55.833986 6926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:39:55.839335 6932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.839351 6934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:39:55.839370 6931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:39:55.839558 6926 server_base.cc:1034] running on GCE node
I20260809 16:39:55.839830 6926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:39:55.840035 6926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:39:55.841171 6926 hybrid_clock.cc:648] HybridClock initialized: now 1786293595841157 us; error 31 us; skew 500 ppm
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293595, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:39:55.843672 6926 init.cc:377] Logged in from keytab as oryx/127.28.62.131@KRBTEST.COM (short username oryx)
I20260809 16:39:55.844077 6926 webserver.cc:460] Webserver started at http://127.28.62.131:42731/ using document root <none> and password file <none>
I20260809 16:39:55.844329 6926 fs_manager.cc:359] Metadata directory not provided
I20260809 16:39:55.844388 6926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:39:55.845858 6926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:55.846495 6943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.846732 6926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:39:55.846804 6926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "f01b369da4fd4cda8e79ea75fe1f7ad2"
format_stamp: "Formatted at 2026-08-09 16:39:55 on dist-test-slave-wl2h"
server_key: "4ba62448fb22d067f0c98700328ecc4c"
server_key_iv: "b6fb5c8c752130d0517b263e39f59f73"
server_key_version: "encryptionkey@0"
I20260809 16:39:55.847234 6926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:39:55.874625 6926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:39:55.876935 6926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:39:55.877097 6926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:39:55.877411 6926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:39:55.877758 6926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:39:55.877813 6926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.877857 6926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:39:55.877884 6926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:39:55.884141 6926 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:34757
I20260809 16:39:55.884191 7081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:34757 every 8 connection(s)
I20260809 16:39:55.884545 6926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.891707 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 6926
W20260809 16:39:55.892572 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.885373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41765 (local address 127.28.62.190:44015)
0809 16:39:55.885441 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.885443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.885711 (+ 268us) server_negotiation.cc:407] Connection header received
0809 16:39:55.886182 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.886183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.886194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.886254 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.887194 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.887690 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.888720 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.888907 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.888962 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.892485 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:41765: BlockingRecv error: recv got EOF from 127.28.62.131:41765 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:39:55.892650 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.885356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:41765)
0809 16:39:55.885520 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.885589 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.886089 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.886306 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.886309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.886886 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.887100 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.887104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.887805 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.887806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.888638 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.888640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.889004 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.889005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.889123 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.889629 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.889637 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.892399 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260809 16:39:55.892872 7082 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:44015 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.62.190:44015: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.897720 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.893178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:41777)
0809 16:39:55.893222 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.893237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.893303 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.893431 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.893432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.893574 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.893619 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.893620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.894246 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.894247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.894761 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.894763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.895201 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.895202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.895277 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.895799 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.895804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.897598 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 16:39:55.897747 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.893244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41777 (local address 127.28.62.190:44015)
0809 16:39:55.893307 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.893309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.893318 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:39:55.893360 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.893360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.893370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.893426 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.893705 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.894163 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.894860 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.895125 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.895202 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.897703 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:41777: BlockingRecv error: recv got EOF from 127.28.62.131:41777 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 16:39:55.900097 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.892363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.28.62.190:44015)
0809 16:39:55.892415 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.892417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.892470 (+ 53us) server_negotiation.cc:407] Connection header received
0809 16:39:55.892543 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.892544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.892557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.892630 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.893514 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.894012 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.894809 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.895004 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.895115 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.896457 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:39:55.896462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:39:55.896462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:39:55.896472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:39:55.898647 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.898999 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.899003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.899003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.899162 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:39:55.899382 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:39:55.899383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:39:55.899383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:39:55.899626 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:39:55.899778 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 16:39:55.899887 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:39:55.900000 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:55 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:55.902427 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:55.898074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:49285)
0809 16:39:55.898184 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:55.898195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:55.898242 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:55.898392 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:55.898392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:55.898575 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:55.898611 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.898612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.899297 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.899298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.899781 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:55.899783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.900172 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.900173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:55.900225 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.900642 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:55.900647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:55.902347 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 16:39:55.902474 6769 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:55.898151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:49285 (local address 127.28.62.190:44015)
0809 16:39:55.898187 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:55.898189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:55.898227 (+ 38us) server_negotiation.cc:407] Connection header received
0809 16:39:55.898305 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:55.898306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:55.898315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:55.898387 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:55.898691 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.899217 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.899863 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:55.900115 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:55.900173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:55.902416 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:49285: BlockingRecv error: recv got EOF from 127.28.62.131:49285 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 16:39:55.902599 7082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:56.626210 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:56.621424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:58511)
0809 16:39:56.621595 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:56.621607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:56.621666 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:56.621801 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:56.621801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:56.621966 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:56.622017 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.622019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.622659 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.622660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.623294 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.623296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.623650 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.623650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.623689 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.624117 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:56.624121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:56.626136 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
W20260809 16:39:56.626368 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:56.621541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:58511 (local address 127.28.62.190:44015)
0809 16:39:56.621676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:56.621678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:56.621686 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:56.621710 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:56.621711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:56.621721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:56.621780 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:56.622083 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.622592 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.623368 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.623568 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.623638 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.626263 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:58511: BlockingRecv error: recv got EOF from 127.28.62.129:58511 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:56.763723 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:56.758504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:60335)
0809 16:39:56.758668 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:56.758689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:56.758747 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:56.758882 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:56.758882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:56.759206 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:56.759259 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.759261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.759988 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.759989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.760669 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.760672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.761186 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.761188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.761261 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.761810 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:56.761816 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:56.763656 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
W20260809 16:39:56.763881 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:56.758660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60335 (local address 127.28.62.190:44015)
0809 16:39:56.758737 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:56.758739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:56.758751 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:39:56.758788 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:56.758789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:56.758800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:56.758863 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:56.759361 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.759886 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.760755 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.761040 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.761153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.763802 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:60335: BlockingRecv error: recv got EOF from 127.28.62.130:60335 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:56 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:56.908406 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:56.903158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:38663)
0809 16:39:56.903310 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:56.903321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:56.903382 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:56.903543 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:56.903544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:56.903710 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:56.903761 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.903763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.904474 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.904476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.905200 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:56.905203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.905668 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.905669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:56.905720 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.906236 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:56.906240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:56.908292 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260809 16:39:56.908448 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:56.903240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:38663 (local address 127.28.62.190:44015)
0809 16:39:56.903332 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:56.903334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:56.903370 (+ 36us) server_negotiation.cc:407] Connection header received
0809 16:39:56.903442 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:56.903442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:56.903455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:56.903524 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:56.903836 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.904345 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.905325 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:56.905565 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:56.905646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:56.908382 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:38663: BlockingRecv error: recv got EOF from 127.28.62.131:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:57.631512 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:57.626907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:59861)
0809 16:39:57.627132 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:57.627143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:57.627200 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:57.627345 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:57.627346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:57.627525 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:57.627584 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.627586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.628280 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.628281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.628818 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.628820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.629153 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.629154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.629195 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.629643 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:57.629647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:57.631452 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":131,"threads_started":1}
W20260809 16:39:57.631639 7092 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:57.626981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59861 (local address 127.28.62.190:44015)
0809 16:39:57.627177 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:57.627179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:57.627189 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:57.627252 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:57.627252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:57.627265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:57.627328 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:57.627654 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.628167 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.628897 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.629083 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.629135 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.631551 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:59861: BlockingRecv error: recv got EOF from 127.28.62.129:59861 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":139,"threads_started":1}
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:57.769455 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:57.764232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:34045)
0809 16:39:57.764362 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:57.764373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:57.764440 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:57.764574 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:57.764574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:57.764748 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:57.764799 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.764800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.765461 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.765462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.766131 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.766134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.766569 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.766570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.766622 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.767322 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:57.767329 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:57.769391 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260809 16:39:57.769590 7092 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:57.764309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:34045 (local address 127.28.62.190:44015)
0809 16:39:57.764364 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:57.764366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:57.764426 (+ 60us) server_negotiation.cc:407] Connection header received
0809 16:39:57.764501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:57.764501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:57.764513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:57.764556 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:57.764864 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.765356 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.766233 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.766472 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.766532 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.769507 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:34045: BlockingRecv error: recv got EOF from 127.28.62.130:34045 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:57 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:57.913993 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:57.908956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:51669)
0809 16:39:57.909085 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:57.909096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:57.909163 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:57.909318 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:57.909319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:57.909515 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:57.909565 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.909567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.910230 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.910232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.910754 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:57.910756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.911207 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.911209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:57.911270 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.911772 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:57.911778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:57.913871 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260809 16:39:57.914247 7092 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:57.909054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:51669 (local address 127.28.62.190:44015)
0809 16:39:57.909098 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:57.909101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:57.909144 (+ 43us) server_negotiation.cc:407] Connection header received
0809 16:39:57.909233 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:57.909233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:57.909245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:57.909296 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:57.909642 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.910153 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.910836 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:57.911083 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:57.911148 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:57.914146 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:51669: BlockingRecv error: recv got EOF from 127.28.62.131:51669 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:58.641687 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:58.632028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:59043)
0809 16:39:58.632210 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:58.632223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:58.632285 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:58.632481 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:58.632482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:58.632675 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:58.632723 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.632725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.633448 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.633449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.633994 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.633996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.634385 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.634386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.634449 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.639504 (+ 5055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:58.639510 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:58.641577 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
W20260809 16:39:58.641760 7097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:58.632221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59043 (local address 127.28.62.190:44015)
0809 16:39:58.632360 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:58.632362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:58.632370 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:39:58.632395 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:58.632396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:58.632410 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:58.632473 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:58.632813 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.633355 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.634087 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.634305 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.634385 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.641685 (+ 7300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:59043: BlockingRecv error: recv got EOF from 127.28.62.129:59043 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:58.774827 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:58.769980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:49099)
0809 16:39:58.770235 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:58.770247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:58.770346 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:58.770504 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:58.770505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:58.770699 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:58.770747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.770748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.771470 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.771471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.772013 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.772015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.772433 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.772434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.772490 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.772937 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:58.772941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:58.774751 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
W20260809 16:39:58.774957 7097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:58.770071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:49099 (local address 127.28.62.190:44015)
0809 16:39:58.770213 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:58.770215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:58.770324 (+ 109us) server_negotiation.cc:407] Connection header received
0809 16:39:58.770408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:58.770409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:58.770422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:58.770495 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:58.770833 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.771389 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.772132 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.772341 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.772433 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.774883 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:49099: BlockingRecv error: recv got EOF from 127.28.62.130:49099 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:58 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:58.919852 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:58.914604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:54825)
0809 16:39:58.914817 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:58.914829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:58.914924 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:58.915139 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:58.915140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:58.915340 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:58.915391 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.915393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.916073 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.916074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.916604 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:58.916606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.916973 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.916974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:58.917055 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.917728 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:58.917734 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:58.919779 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
W20260809 16:39:58.920001 7097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:58.914724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:54825 (local address 127.28.62.190:44015)
0809 16:39:58.914803 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:58.914806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:58.914903 (+ 97us) server_negotiation.cc:407] Connection header received
0809 16:39:58.915007 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:58.915032 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:58.915055 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:58.915117 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:58.915478 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.915982 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.916699 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:58.916895 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:58.916975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:58.919913 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:54825: BlockingRecv error: recv got EOF from 127.28.62.131:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:59.647706 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:59.642423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:50825)
0809 16:39:59.642621 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:59.642638 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:59.642713 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:59.642872 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:59.642872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:59.643160 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:59.643228 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.643230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.643975 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.643976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.644850 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.644853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.645189 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.645190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.645234 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.645775 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:59.645780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:59.647645 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
W20260809 16:39:59.647818 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:59.642509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:50825 (local address 127.28.62.190:44015)
0809 16:39:59.642699 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:59.642701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:59.642711 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:39:59.642766 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:59.642767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:59.642778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:59.642839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:59.643342 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.643876 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.644920 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.645115 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.645173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.647738 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:50825: BlockingRecv error: recv got EOF from 127.28.62.129:50825 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:59.780805 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:59.775424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:51911)
0809 16:39:59.775610 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:59.775630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:59.775707 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:59.775883 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:59.775884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:59.776056 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:59.776138 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.776141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.776858 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.776860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.777745 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.777749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.778179 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.778180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.778238 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.778761 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:59.778766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:59.780744 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":130,"threads_started":1}
W20260809 16:39:59.780977 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:59.775527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:51911 (local address 127.28.62.190:44015)
0809 16:39:59.775605 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:59.775608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:59.775759 (+ 151us) server_negotiation.cc:407] Connection header received
0809 16:39:59.775795 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:59.775796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:59.775809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:59.775856 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:59.776268 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.776770 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.777854 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.778081 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.778145 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.780903 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:51911: BlockingRecv error: recv got EOF from 127.28.62.130:51911 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:39:59 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:39:59.924863 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:39:59.920386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:43229)
0809 16:39:59.920536 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 16:39:59.920547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:39:59.920607 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:39:59.920720 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:39:59.920720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:39:59.920884 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:39:59.920934 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.920936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.921570 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.921571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.922098 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:39:59.922100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.922470 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.922471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:39:59.922508 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.922954 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:39:59.922958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:39:59.924763 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260809 16:39:59.924914 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:39:59.920470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:43229 (local address 127.28.62.190:44015)
0809 16:39:59.920553 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:39:59.920555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:39:59.920595 (+ 40us) server_negotiation.cc:407] Connection header received
0809 16:39:59.920651 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:39:59.920652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:39:59.920665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:39:59.920707 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:39:59.920994 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.921494 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.922196 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:39:59.922386 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:39:59.922462 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:39:59.924846 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:43229: BlockingRecv error: recv got EOF from 127.28.62.131:43229 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:00.653102 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:00.648225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:56685)
0809 16:40:00.648386 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:00.648397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:00.648449 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:00.648614 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:00.648614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:00.648793 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:00.648847 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.648849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.649524 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.649524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.650069 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.650071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.650461 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.650462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.650514 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.651235 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:00.651241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:00.653040 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260809 16:40:00.653275 7105 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:00.648304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:56685 (local address 127.28.62.190:44015)
0809 16:40:00.648440 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:00.648442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:00.648453 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:40:00.648510 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:00.648511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:00.648522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:00.648576 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:00.648914 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.649416 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.650178 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.650376 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.650431 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.653175 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:56685: BlockingRecv error: recv got EOF from 127.28.62.129:56685 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:00.786015 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:00.781331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:58683)
0809 16:40:00.781494 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:00.781505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:00.781555 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:00.781704 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:00.781705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:00.781876 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:00.781927 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.781928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.782648 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.782660 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.783236 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.783238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.783589 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.783590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.783628 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.784092 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:00.784097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:00.785913 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
W20260809 16:40:00.786064 7105 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:00.781468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:58683 (local address 127.28.62.190:44015)
0809 16:40:00.781537 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:00.781539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:00.781548 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:40:00.781605 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:00.781606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:00.781617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:00.781672 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:00.782021 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.782568 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.783323 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.783520 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.783581 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.785998 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:58683: BlockingRecv error: recv got EOF from 127.28.62.130:58683 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:00 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:00.940369 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:00.926333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:51439)
0809 16:40:00.926732 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:00.926745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:00.926910 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:00.927483 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:00.927484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:00.927990 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:00.928149 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.928159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.931048 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.931049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.933017 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:00.933028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.934338 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.934339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:00.934390 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.935833 (+ 1443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:00.935839 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:00.940158 (+ 4319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
W20260809 16:40:00.940464 7105 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:00.926652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:51439 (local address 127.28.62.190:44015)
0809 16:40:00.926855 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:00.926857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:00.926900 (+ 43us) server_negotiation.cc:407] Connection header received
0809 16:40:00.927158 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:00.927159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:00.927174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:00.927350 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:00.928598 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.930780 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.933307 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:00.934056 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:00.934303 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:00.940341 (+ 6038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:51439: BlockingRecv error: recv got EOF from 127.28.62.131:51439 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:01.658952 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:01.653686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:58259)
0809 16:40:01.653862 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:01.653873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:01.653926 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:01.654083 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:01.654084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:01.654274 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:01.654327 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.654329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.655108 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.655108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.655668 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.655669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.656045 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.656046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.656084 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.656604 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:01.656609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:01.658890 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
W20260809 16:40:01.659205 7110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:01.653804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:58259 (local address 127.28.62.190:44015)
0809 16:40:01.653942 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:01.653944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:01.653953 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:40:01.653979 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:01.653980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:01.653992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:01.654069 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:01.654432 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.655035 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.655761 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.655972 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.656033 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.659119 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:58259: BlockingRecv error: recv got EOF from 127.28.62.129:58259 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:01.792263 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:01.786624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:43845)
0809 16:40:01.786797 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:01.786810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:01.786885 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:01.787092 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:01.787094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:01.787292 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:01.787343 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.787344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.788101 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.788102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.788728 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.788730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.789189 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.789190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.789239 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.789999 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:01.790004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:01.792160 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
W20260809 16:40:01.792316 7110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:01.786735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:43845 (local address 127.28.62.190:44015)
0809 16:40:01.786796 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:01.786798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:01.786881 (+ 83us) server_negotiation.cc:407] Connection header received
0809 16:40:01.786939 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:01.786940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:01.786950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:01.787045 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:01.787445 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.787990 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.788845 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.789082 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.789149 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.792251 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:43845: BlockingRecv error: recv got EOF from 127.28.62.130:43845 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:01 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:01.945961 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:01.941075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:56121)
0809 16:40:01.941300 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:01.941313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:01.941418 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:01.941593 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:01.941594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:01.941796 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:01.941845 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.941846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.942529 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.942530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.943109 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:01.943111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.943504 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.943505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:01.943564 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.944032 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:01.944036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:01.945863 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260809 16:40:01.946010 7110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:01.941186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:56121 (local address 127.28.62.190:44015)
0809 16:40:01.941289 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:01.941292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:01.941398 (+ 106us) server_negotiation.cc:407] Connection header received
0809 16:40:01.941490 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:01.941490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:01.941512 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:01.941583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:01.941929 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.942439 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.943200 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:01.943396 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:01.943502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:01.945944 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:56121: BlockingRecv error: recv got EOF from 127.28.62.131:56121 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:02.664481 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:02.659509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:41763)
0809 16:40:02.659691 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:02.659702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:02.659764 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:02.659908 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:02.659908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:02.660079 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:02.660129 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.660131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.660793 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.660794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.661327 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.661328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.661815 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.661816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.661891 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.662424 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:02.662429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:02.664418 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260809 16:40:02.664641 7114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:02.659631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:41763 (local address 127.28.62.190:44015)
0809 16:40:02.659770 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:02.659772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:02.659785 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:40:02.659819 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:02.659820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:02.659831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:02.659890 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:02.660208 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.660717 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.661466 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.661717 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.661812 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.664548 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:41763: BlockingRecv error: recv got EOF from 127.28.62.129:41763 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:02.798169 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:02.792967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:53329)
0809 16:40:02.793112 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:02.793126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:02.793286 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:02.793483 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:02.793485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:02.793701 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:02.793751 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.793753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.794479 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.794480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.795104 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.795106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.795482 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.795483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.795541 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.796065 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:02.796070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:02.798107 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260809 16:40:02.798321 7114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:02.793060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:53329 (local address 127.28.62.190:44015)
0809 16:40:02.793116 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:02.793118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:02.793206 (+ 88us) server_negotiation.cc:407] Connection header received
0809 16:40:02.793364 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:02.793365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:02.793381 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:02.793461 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:02.793836 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.794377 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.795207 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.795405 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.795485 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.798236 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:53329: BlockingRecv error: recv got EOF from 127.28.62.130:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:02 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:02.952008 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:02.946619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:36527)
0809 16:40:02.946783 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:02.946801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:02.946900 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:02.947155 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:02.947156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:02.947336 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:02.947384 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.947386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.948079 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.948080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.948608 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:02.948610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.948980 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.948981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:02.949048 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.949574 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:02.949579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:02.951948 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260809 16:40:02.952270 7114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:02.946707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:36527 (local address 127.28.62.190:44015)
0809 16:40:02.946808 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:02.946810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:02.946887 (+ 77us) server_negotiation.cc:407] Connection header received
0809 16:40:02.946962 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:02.946963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:02.946980 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:02.947068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:02.947472 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.947981 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.948697 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:02.948900 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:02.948981 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:02.952222 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:36527: BlockingRecv error: recv got EOF from 127.28.62.131:36527 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:03.675413 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:03.665023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:42081)
0809 16:40:03.665226 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:03.665237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:03.665288 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:03.665422 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:03.665422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:03.665589 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:03.665639 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.665640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.666329 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.666330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.666864 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.666866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.667301 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.667302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.667340 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.673320 (+ 5980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:03.673326 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:03.675351 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":134,"threads_started":1}
W20260809 16:40:03.675562 7118 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:03.665148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:42081 (local address 127.28.62.190:44015)
0809 16:40:03.665275 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:03.665276 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:03.665286 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:40:03.665334 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:03.665335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:03.665349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:03.665413 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:03.665734 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.666256 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.666956 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.667223 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.667284 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.675466 (+ 8182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:42081: BlockingRecv error: recv got EOF from 127.28.62.129:42081 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:03.804119 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:03.798715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:56729)
0809 16:40:03.798881 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:03.798894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:03.798964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:03.799250 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:03.799251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:03.799424 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:03.799475 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.799477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.800373 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.800374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.801173 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.801176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.801614 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.801615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.801665 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.802162 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:03.802167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:03.804050 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
W20260809 16:40:03.804283 7118 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:03.798831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:56729 (local address 127.28.62.190:44015)
0809 16:40:03.798907 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:03.798909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:03.798951 (+ 42us) server_negotiation.cc:407] Connection header received
0809 16:40:03.799052 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:03.799053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:03.799070 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:03.799205 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:03.799584 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.800275 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.801232 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.801490 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.801545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.804190 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:56729: BlockingRecv error: recv got EOF from 127.28.62.130:56729 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:03 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:03.957683 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:03.952544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:41567)
0809 16:40:03.952746 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:03.952757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:03.952855 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:03.953034 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:03.953035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:03.953274 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:03.953325 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.953327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.954164 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.954165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.954705 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:03.954707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.955111 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.955112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:03.955171 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.955681 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:03.955685 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:03.957618 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
W20260809 16:40:03.957849 7118 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:03.952618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41567 (local address 127.28.62.190:44015)
0809 16:40:03.952696 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:03.952699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:03.952827 (+ 128us) server_negotiation.cc:407] Connection header received
0809 16:40:03.952914 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:03.952915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:03.952930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:03.953010 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:03.953413 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.954081 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.954799 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:03.955049 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:03.955119 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:03.957752 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:41567: BlockingRecv error: recv got EOF from 127.28.62.131:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:04.681339 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:04.675954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:38447)
0809 16:40:04.676103 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:04.676115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:04.676174 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:04.676485 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:04.676486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:04.676723 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:04.676785 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.676787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.677566 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.677567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.678166 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.678167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.678595 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.678596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.678650 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.679195 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:04.679201 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:04.681240 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260809 16:40:04.681459 7123 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:04.676049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:38447 (local address 127.28.62.190:44015)
0809 16:40:04.676266 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:04.676269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:04.676280 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:40:04.676312 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:04.676313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:04.676325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:04.676457 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:04.676856 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.677460 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.678273 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.678480 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.678546 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.681367 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:38447: BlockingRecv error: recv got EOF from 127.28.62.129:38447 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:04.810321 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:04.804815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:56211)
0809 16:40:04.804993 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:04.805006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:04.805093 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:04.805288 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:04.805289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:04.805493 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:04.805573 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.805576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.806418 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.806419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.806954 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.806956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.807393 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.807394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.807441 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.807948 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:04.807954 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:04.810246 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
W20260809 16:40:04.810518 7123 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:04.804917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:56211 (local address 127.28.62.190:44015)
0809 16:40:04.805023 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:04.805024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:04.805092 (+ 68us) server_negotiation.cc:407] Connection header received
0809 16:40:04.805151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:04.805152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:04.805167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:04.805268 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:04.805666 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.806331 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.807059 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.807298 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.807382 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.810433 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:56211: BlockingRecv error: recv got EOF from 127.28.62.130:56211 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:04 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:04.963584 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:04.958317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:48487)
0809 16:40:04.958520 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:04.958532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:04.958657 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:04.958849 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:04.958850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:04.959111 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:04.959176 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.959178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.959869 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.959870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.960418 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:04.960420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.960821 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.960822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:04.960884 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.961419 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:04.961425 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:04.963476 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
W20260809 16:40:04.963636 7123 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:04.958534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:48487 (local address 127.28.62.190:44015)
0809 16:40:04.958616 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:04.958618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:04.958633 (+ 15us) server_negotiation.cc:407] Connection header received
0809 16:40:04.958752 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:04.958753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:04.958773 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:04.958841 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:04.959268 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.959768 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.960514 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:04.960709 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:04.960819 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:04.963568 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:48487: BlockingRecv error: recv got EOF from 127.28.62.131:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:05.687053 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:05.681930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:59319)
0809 16:40:05.682090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:05.682107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:05.682161 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:05.682348 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:05.682349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:05.682528 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:05.682579 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.682581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.683305 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.683306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.683833 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.683835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.684277 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.684279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.684331 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.684909 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:05.684913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:05.686931 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
W20260809 16:40:05.687240 7127 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:05.682044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59319 (local address 127.28.62.190:44015)
0809 16:40:05.682182 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:05.682184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:05.682195 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:40:05.682233 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:05.682234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:05.682246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:05.682315 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:05.682672 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.683224 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.683964 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.684172 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.684240 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.687158 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:59319: BlockingRecv error: recv got EOF from 127.28.62.129:59319 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":132,"threads_started":1}
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:05.815980 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:05.810840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:44487)
0809 16:40:05.810987 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:05.810998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:05.811101 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:05.811240 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:05.811241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:05.811412 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:05.811462 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.811464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.812264 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.812265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.812814 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.812815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.813247 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.813248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.813316 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.813818 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:05.813823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:05.815900 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260809 16:40:05.816161 7127 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:05.810976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:44487 (local address 127.28.62.190:44015)
0809 16:40:05.811066 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:05.811068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:05.811077 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:40:05.811152 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:05.811153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:05.811163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:05.811222 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:05.811560 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.812156 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.812926 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.813171 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.813237 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.816030 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:44487: BlockingRecv error: recv got EOF from 127.28.62.130:44487 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:05 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:05.969481 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:05.964112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:46835)
0809 16:40:05.964387 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:05.964397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:05.964478 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:05.964649 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:05.964651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:05.964840 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:05.964889 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.964892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.965601 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.965602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.966131 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:05.966133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.966506 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.966507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:05.966575 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.967193 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:05.967198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:05.969420 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":218,"threads_started":1}
W20260809 16:40:05.969681 7127 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:05.964264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:46835 (local address 127.28.62.190:44015)
0809 16:40:05.964329 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:05.964332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:05.964460 (+ 128us) server_negotiation.cc:407] Connection header received
0809 16:40:05.964536 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:05.964537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:05.964553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:05.964642 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:05.964977 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.965498 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.966230 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:05.966428 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:05.966508 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:05.969608 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:46835: BlockingRecv error: recv got EOF from 127.28.62.131:46835 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:06.692576 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:06.687605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:49047)
0809 16:40:06.687771 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:06.687791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:06.687846 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:06.688002 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:06.688003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:06.688196 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:06.688250 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.688252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.688934 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.688935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.689501 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.689502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.689890 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.689891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.689938 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.690546 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:06.690551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:06.692498 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260809 16:40:06.692728 7134 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:06.687720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:49047 (local address 127.28.62.190:44015)
0809 16:40:06.687856 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:06.687858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:06.687891 (+ 33us) server_negotiation.cc:407] Connection header received
0809 16:40:06.687913 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:06.687914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:06.687925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:06.687982 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:06.688339 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.688859 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.689592 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.689807 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.689871 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.692609 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:49047: BlockingRecv error: recv got EOF from 127.28.62.129:49047 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:06.821451 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:06.816526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:53195)
0809 16:40:06.816688 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:06.816707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:06.816768 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:06.816946 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:06.816948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:06.817212 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:06.817264 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.817266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.817985 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.817986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.818517 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.818519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.818906 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.818907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.818949 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.819523 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:06.819528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:06.821381 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
W20260809 16:40:06.821626 7134 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:06.816616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:53195 (local address 127.28.62.190:44015)
0809 16:40:06.816665 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:06.816667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:06.816774 (+ 107us) server_negotiation.cc:407] Connection header received
0809 16:40:06.816826 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:06.816827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:06.816838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:06.816882 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:06.817377 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.817890 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.818617 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.818825 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.818896 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.821543 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:53195: BlockingRecv error: recv got EOF from 127.28.62.130:53195 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:06 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:06.974967 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:06.970105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:51387)
0809 16:40:06.970272 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:06.970289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:06.970349 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:06.970494 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:06.970494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:06.970675 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:06.970725 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.970727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.971484 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.971486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.972015 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:06.972017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.972398 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.972399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:06.972451 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.973000 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:06.973005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:06.974882 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
W20260809 16:40:06.975255 7134 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:06.970186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:51387 (local address 127.28.62.190:44015)
0809 16:40:06.970252 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:06.970254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:06.970358 (+ 104us) server_negotiation.cc:407] Connection header received
0809 16:40:06.970397 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:06.970398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:06.970412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:06.970468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:06.970808 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.971400 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.972108 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:06.972305 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:06.972358 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:06.975165 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:51387: BlockingRecv error: recv got EOF from 127.28.62.131:51387 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:07.698568 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:07.693154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:45383)
0809 16:40:07.693365 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:07.693378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:07.693448 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:07.693681 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:07.693682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:07.693863 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:07.693914 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.693917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.694723 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.694724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.695386 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.695389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.695781 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.695782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.695825 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.696337 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:07.696342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:07.698440 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260809 16:40:07.698642 7138 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:07.693309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:45383 (local address 127.28.62.190:44015)
0809 16:40:07.693491 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:07.693493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:07.693503 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:40:07.693527 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:07.693528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:07.693541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:07.693615 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:07.694026 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.694620 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.695483 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.695696 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.695767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.698540 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:45383: BlockingRecv error: recv got EOF from 127.28.62.129:45383 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:07.827219 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:07.822049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:46277)
0809 16:40:07.822266 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:07.822279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:07.822362 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:07.822501 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:07.822502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:07.822712 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:07.822760 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.822762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.823543 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.823545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.824101 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.824102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.824469 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.824470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.824511 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.825118 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:07.825124 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:07.827138 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":134,"threads_started":1}
W20260809 16:40:07.827275 7138 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:07.822215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:46277 (local address 127.28.62.190:44015)
0809 16:40:07.822277 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:07.822279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:07.822336 (+ 57us) server_negotiation.cc:407] Connection header received
0809 16:40:07.822414 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:07.822414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:07.822425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:07.822479 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:07.822850 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.823453 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.824194 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.824393 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.824458 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.827225 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:46277: BlockingRecv error: recv got EOF from 127.28.62.130:46277 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:07 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:07.980683 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:07.975589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:52655)
0809 16:40:07.975728 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:07.975740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:07.975825 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:07.976006 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:07.976007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:07.976199 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:07.976259 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.976261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.977051 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.977052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.977579 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:07.977580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.977993 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.977994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:07.978048 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.978521 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:07.978526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:07.980585 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260809 16:40:07.980747 7138 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:07.975675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:52655 (local address 127.28.62.190:44015)
0809 16:40:07.975769 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:07.975771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:07.975809 (+ 38us) server_negotiation.cc:407] Connection header received
0809 16:40:07.975898 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:07.975899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:07.975914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:07.975979 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:07.976340 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.976982 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.977660 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:07.977856 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:07.977956 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:07.980681 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:52655: BlockingRecv error: recv got EOF from 127.28.62.131:52655 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:08.709530 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:08.699183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:36505)
0809 16:40:08.699313 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:08.699327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:08.699405 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:08.699542 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:08.699543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:08.699725 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:08.699775 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.699777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.700460 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.700461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.701020 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.701022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.701401 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.701402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.701450 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.707337 (+ 5887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:08.707343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:08.709466 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
W20260809 16:40:08.709694 7144 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:08.699270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:36505 (local address 127.28.62.190:44015)
0809 16:40:08.699411 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:08.699413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:08.699423 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:40:08.699450 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:08.699451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:08.699462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:08.699520 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:08.699845 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.700384 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.701103 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.701329 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.701386 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.709594 (+ 8208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:36505: BlockingRecv error: recv got EOF from 127.28.62.129:36505 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:08.832846 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:08.827826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:51393)
0809 16:40:08.828002 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:08.828012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:08.828101 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:08.828267 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:08.828268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:08.828463 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:08.828509 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.828511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.829192 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.829194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.829726 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.829728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.830094 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.830095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.830153 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.830656 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:08.830661 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:08.832740 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
W20260809 16:40:08.832901 7144 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:08.827998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:51393 (local address 127.28.62.190:44015)
0809 16:40:08.828090 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:08.828092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:08.828103 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:40:08.828167 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:08.828167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:08.828183 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:08.828260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:08.828599 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.829101 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.829829 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.830020 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.830097 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.832832 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:51393: BlockingRecv error: recv got EOF from 127.28.62.130:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:08 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:08.986567 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:08.981472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:43431)
0809 16:40:08.981608 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:08.981627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:08.981694 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:08.981881 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:08.981882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:08.982068 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:08.982118 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.982120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.982811 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.982812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.983478 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:08.983481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.983921 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.983922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:08.983976 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.984592 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:08.984597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:08.986485 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260809 16:40:08.986668 7144 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:08.981565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:43431 (local address 127.28.62.190:44015)
0809 16:40:08.981648 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:08.981650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:08.981746 (+ 96us) server_negotiation.cc:407] Connection header received
0809 16:40:08.981779 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:08.981779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:08.981794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:08.981854 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:08.982192 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.982715 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.983627 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:08.983822 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:08.983890 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:08.986597 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:43431: BlockingRecv error: recv got EOF from 127.28.62.131:43431 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:09.714891 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:09.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:32997)
0809 16:40:09.710293 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:09.710305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:09.710366 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:09.710508 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:09.710509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:09.710684 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:09.710740 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.710742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.711454 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.711455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.711984 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.711986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.712358 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.712359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.712401 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.712948 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:09.712953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:09.714830 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
W20260809 16:40:09.715082 7149 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:09.710219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:32997 (local address 127.28.62.190:44015)
0809 16:40:09.710336 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:09.710337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:09.710350 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:40:09.710413 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:09.710414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:09.710424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:09.710483 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:09.710810 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.711375 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.712087 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.712285 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.712344 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.714964 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:32997: BlockingRecv error: recv got EOF from 127.28.62.129:32997 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:09.838294 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:09.833445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:44093)
0809 16:40:09.833650 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:09.833662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:09.833737 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:09.833907 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:09.833908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:09.834106 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:09.834154 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.834156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.834829 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.834829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.835459 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.835461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.835783 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.835784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.835826 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.836278 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:09.836282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:09.838173 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":158,"threads_started":1}
W20260809 16:40:09.838326 7149 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:09.833522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:44093 (local address 127.28.62.190:44015)
0809 16:40:09.833589 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:09.833591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:09.833798 (+ 207us) server_negotiation.cc:407] Connection header received
0809 16:40:09.833828 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:09.833829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:09.833843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:09.833900 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:09.834221 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.834749 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.835532 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.835711 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.835771 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.838260 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:44093: BlockingRecv error: recv got EOF from 127.28.62.130:44093 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:09 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.131@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:09.992281 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:09.987130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.131:44697)
0809 16:40:09.987341 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:09.987355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:09.987435 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:09.987594 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:09.987595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:09.987766 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:09.987813 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.987815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.988522 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.988523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.989044 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:09.989045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.989563 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.989564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:09.989614 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.990166 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:09.990170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:09.992147 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":123,"threads_started":1}
W20260809 16:40:09.992286 7149 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:09.987302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:44697 (local address 127.28.62.190:44015)
0809 16:40:09.987379 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:09.987382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:09.987419 (+ 37us) server_negotiation.cc:407] Connection header received
0809 16:40:09.987506 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:09.987507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:09.987537 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:09.987590 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:09.987890 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.988412 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.989266 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:09.989466 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:09.989535 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:09.992231 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.131:44697: BlockingRecv error: recv got EOF from 127.28.62.131:44697 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 09 16:40:10 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:10 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.129@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:10.720523 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:10.715485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.129:47711)
0809 16:40:10.715636 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:10.715657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:10.715725 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:10.715916 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:10.715917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:10.716115 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:10.716171 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:10.716174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.716862 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.716863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:10.717497 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:10.717500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.717879 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.717880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:10.717939 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:10.718436 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:10.718441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:10.720454 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260809 16:40:10.720677 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:10.715548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47711 (local address 127.28.62.190:44015)
0809 16:40:10.715693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:10.715696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:10.715705 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:40:10.715807 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:10.715808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:10.715822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:10.715907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:10.716258 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.716770 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.717596 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.717805 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.717879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:10.720574 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.129:47711: BlockingRecv error: recv got EOF from 127.28.62.129:47711 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
Aug 09 16:40:10 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
Aug 09 16:40:10 dist-test-slave-wl2h krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.62.130@KRBTEST.COM for oryx/127.28.62.190@KRBTEST.COM, Server not found in Kerberos database
I20260809 16:40:10.843792 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:10.838897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44015 (local address 127.28.62.130:43611)
0809 16:40:10.839164 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:10.839177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:10.839245 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:10.839406 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:10.839408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:10.839602 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:10.839664 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:10.839666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.840361 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.840363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:10.840882 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:10.840884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.841252 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.841253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:10.841311 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:10.841817 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:10.841822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:10.843727 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.62.190:44015: Server oryx/127.28.62.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"spinlock_wait_cycles":17664,"thread_start_us":152,"threads_started":1}
W20260809 16:40:10.843976 7154 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:10.839118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:43611 (local address 127.28.62.190:44015)
0809 16:40:10.839235 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:10.839237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:10.839249 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:40:10.839305 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:10.839306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:10.839320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:10.839398 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:10.839751 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.840270 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.840969 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:10.841174 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:10.841253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:10.843887 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.62.130:43611: BlockingRecv error: recv got EOF from 127.28.62.130:43611 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 16:40:10.900696 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6589
I20260809 16:40:10.909689 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6770
I20260809 16:40:10.917060 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6926
I20260809 16:40:10.923964 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 6510
2026-08-09T16:40:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17749 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T16:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:40:10Z Disabled control of system clock
I20260809 16:40:10.955523 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:33687
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:33687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 16:40:11.042466 7160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.042693 7160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.042714 7160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.043926 7160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:40:11.043967 7160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.043979 7160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.043992 7160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:40:11.044004 7160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:40:11.045288 7160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:33687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:33687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.045488 7160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.045746 7160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.050685 7167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.050669 7166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.050776 7160 server_base.cc:1034] running on GCE node
W20260809 16:40:11.050669 7169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.051225 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.051483 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.052626 7160 hybrid_clock.cc:648] HybridClock initialized: now 1786293611052612 us; error 32 us; skew 500 ppm
I20260809 16:40:11.054013 7160 webserver.cc:460] Webserver started at http://127.28.62.190:42863/ using document root <none> and password file <none>
I20260809 16:40:11.054311 7160 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.054390 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.054550 7160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.055701 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "2a19dbc757ba432eaf42f9f9a1939205"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "a83c2d7df4c39ae1777d88bc0e48f0c4"
server_key_iv: "085048c62f6f06d60ab7520f9a115574"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.056190 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "2a19dbc757ba432eaf42f9f9a1939205"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "a83c2d7df4c39ae1777d88bc0e48f0c4"
server_key_iv: "085048c62f6f06d60ab7520f9a115574"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.057560 7160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:40:11.058283 7175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.058462 7160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:40:11.058580 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "2a19dbc757ba432eaf42f9f9a1939205"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "a83c2d7df4c39ae1777d88bc0e48f0c4"
server_key_iv: "085048c62f6f06d60ab7520f9a115574"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.058665 7160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.076941 7160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.077302 7160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.077467 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.081573 7160 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:33687
I20260809 16:40:11.081645 7235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:33687 every 8 connection(s)
I20260809 16:40:11.081997 7160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:40:11.082692 7236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:11.085424 7236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: Bootstrap starting.
I20260809 16:40:11.086037 7236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:11.086352 7236 log.cc:824] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:11.087081 7236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: No bootstrap required, opened a new log
I20260809 16:40:11.088655 7236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } }
I20260809 16:40:11.088809 7236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:11.088833 7236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a19dbc757ba432eaf42f9f9a1939205, State: Initialized, Role: FOLLOWER
I20260809 16:40:11.089012 7236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } }
I20260809 16:40:11.089077 7236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:40:11.089120 7236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:40:11.089170 7236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:11.089708 7236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } }
I20260809 16:40:11.089821 7236 leader_election.cc:302] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a19dbc757ba432eaf42f9f9a1939205; no voters:
I20260809 16:40:11.090014 7236 leader_election.cc:288] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:40:11.090154 7239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:40:11.090324 7236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:40:11.090405 7239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [term 1 LEADER]: Becoming Leader. State: Replica: 2a19dbc757ba432eaf42f9f9a1939205, State: Running, Role: LEADER
I20260809 16:40:11.090576 7239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } }
I20260809 16:40:11.090730 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7160
I20260809 16:40:11.090827 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:40:11.091089 28922 external_mini_cluster.cc:1428] Setting key 82160757dee9b0cb5d57a2962462daee
I20260809 16:40:11.091379 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a19dbc757ba432eaf42f9f9a1939205. Latest consensus state: current_term: 1 leader_uuid: "2a19dbc757ba432eaf42f9f9a1939205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } } }
I20260809 16:40:11.091493 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [sys.catalog]: This master's current role is: LEADER
I20260809 16:40:11.091535 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a19dbc757ba432eaf42f9f9a1939205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a19dbc757ba432eaf42f9f9a1939205" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33687 } } }
I20260809 16:40:11.091626 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205 [sys.catalog]: This master's current role is: LEADER
W20260809 16:40:11.092514 7260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:40:11.092573 7260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:40:11.092648 7251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:40:11.092878 7251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:40:11.095055 7251 catalog_manager.cc:1347] Generated new cluster ID: 31f2810f9014430fa4ace6fb910fbd52
I20260809 16:40:11.095110 7251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:40:11.097179 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.091782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.28.62.190:33687)
0809 16:40:11.092001 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.092101 (+ 100us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.092414 (+ 313us) server_negotiation.cc:407] Connection header received
0809 16:40:11.092551 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.092557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.092648 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.092789 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.093041 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.094162 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.094981 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.095603 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.095717 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.096491 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.096494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.096494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.096701 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.096741 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.096754 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.096817 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.096901 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.096973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260809 16:40:11.101817 7251 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:40:11.102483 7251 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:40:11.113791 7251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a19dbc757ba432eaf42f9f9a1939205: Generated new TSK 0
I20260809 16:40:11.114001 7251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:40:11.115846 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--builtin_ntp_servers=127.28.62.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 16:40:11.215794 7264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.216058 7264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.216100 7264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.217329 7264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.217401 7264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.217446 7264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:40:11.218811 7264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.219125 7264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.219465 7264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.225584 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.225780 7264 server_base.cc:1034] running on GCE node
W20260809 16:40:11.225591 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.225656 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.226151 7264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.226419 7264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.227584 7264 hybrid_clock.cc:648] HybridClock initialized: now 1786293611227558 us; error 47 us; skew 500 ppm
I20260809 16:40:11.228925 7264 webserver.cc:460] Webserver started at http://127.28.62.129:46543/ using document root <none> and password file <none>
I20260809 16:40:11.229245 7264 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.229331 7264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.229475 7264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.230571 7264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "57680cc1007e4105b2450fb0dca22fa1"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7e6f1570926652abea12c114a7a3b736"
server_key_iv: "6493e92144023ce20124a972fd54d723"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.230950 7264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "57680cc1007e4105b2450fb0dca22fa1"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7e6f1570926652abea12c114a7a3b736"
server_key_iv: "6493e92144023ce20124a972fd54d723"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.232414 7264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:11.233251 7279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.233402 7264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:11.233510 7264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "57680cc1007e4105b2450fb0dca22fa1"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7e6f1570926652abea12c114a7a3b736"
server_key_iv: "6493e92144023ce20124a972fd54d723"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.233587 7264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.249284 7264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.249619 7264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.249774 7264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.250018 7264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:11.250385 7264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:11.250444 7264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.250488 7264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:11.250517 7264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.256875 7264 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:37253
I20260809 16:40:11.256922 7411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:37253 every 8 connection(s)
I20260809 16:40:11.257259 7264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:40:11.262771 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.258015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:36893 (local address 127.28.62.190:33687)
0809 16:40:11.258078 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.258080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.258351 (+ 271us) server_negotiation.cc:407] Connection header received
0809 16:40:11.258890 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.258892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.258909 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.258963 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.259697 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.260218 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.261249 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.261544 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.261623 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.262441 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.262442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.262443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.262448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.262463 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.262469 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.262513 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.262624 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.262660 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 16:40:11.262959 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.258009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33687 (local address 127.28.62.129:36893)
0809 16:40:11.258198 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:11.258233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:11.258782 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:11.259086 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:11.259090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:11.259098 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:11.259587 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.259591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.260329 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.260330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.261151 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.261153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.261633 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.261634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.261775 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.262302 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:11.262302 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:11.262330 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:11.262352 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:11.262526 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:11.262532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:11.262536 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:40:11.262615 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 16:40:11.262716 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260809 16:40:11.263103 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7264
I20260809 16:40:11.263216 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:40:11.263481 28922 external_mini_cluster.cc:1428] Setting key 54453f5ab84c7881c038eb3e8d899d1c
I20260809 16:40:11.263549 7412 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33687
I20260809 16:40:11.263643 7412 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:11.263831 7412 heartbeater.cc:511] Master 127.28.62.190:33687 requested a full tablet report, sending...
I20260809 16:40:11.264394 7193 ts_manager.cc:194] Registered new tserver with Master: 57680cc1007e4105b2450fb0dca22fa1 (127.28.62.129:37253)
I20260809 16:40:11.264549 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--builtin_ntp_servers=127.28.62.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 16:40:11.265115 7193 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.129:36893
W20260809 16:40:11.349943 7416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.350195 7416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.350234 7416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.351507 7416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.351575 7416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.351634 7416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:40:11.353061 7416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.353355 7416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.353699 7416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:40:11.359179 7416 server_base.cc:1034] running on GCE node
W20260809 16:40:11.359203 7426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.359198 7422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.359388 7424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.359645 7416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.359969 7416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.361163 7416 hybrid_clock.cc:648] HybridClock initialized: now 1786293611361113 us; error 31 us; skew 500 ppm
I20260809 16:40:11.362612 7416 webserver.cc:460] Webserver started at http://127.28.62.130:39329/ using document root <none> and password file <none>
I20260809 16:40:11.362927 7416 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.362977 7416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.363185 7416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.364349 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "5c9f36628a2846adab33ed568c95244a"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "b938afe4f3b80aca7fe1a954ed65efcc"
server_key_iv: "a92fa2008ff5a1fbee0d0fadef2fb0e9"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.364722 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "5c9f36628a2846adab33ed568c95244a"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "b938afe4f3b80aca7fe1a954ed65efcc"
server_key_iv: "a92fa2008ff5a1fbee0d0fadef2fb0e9"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.366183 7416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:40:11.367183 7434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.367435 7416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:11.367564 7416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "5c9f36628a2846adab33ed568c95244a"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "b938afe4f3b80aca7fe1a954ed65efcc"
server_key_iv: "a92fa2008ff5a1fbee0d0fadef2fb0e9"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.367650 7416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.388088 7416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.388447 7416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.388612 7416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.388880 7416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:11.389252 7416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:11.389312 7416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.389357 7416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:11.389384 7416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.397269 7416 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:42383
I20260809 16:40:11.397313 7566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:42383 every 8 connection(s)
I20260809 16:40:11.397694 7416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:40:11.400591 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7416
I20260809 16:40:11.400683 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:40:11.400902 28922 external_mini_cluster.cc:1428] Setting key 931285ced99220e055cb837ec74fc5e6
I20260809 16:40:11.402002 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--builtin_ntp_servers=127.28.62.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 16:40:11.404033 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.398638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:35643 (local address 127.28.62.190:33687)
0809 16:40:11.398693 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.398695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.398914 (+ 219us) server_negotiation.cc:407] Connection header received
0809 16:40:11.399526 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.399527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.399540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.399621 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.400364 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.400981 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.402265 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.402462 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.402523 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.403568 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.403569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.403569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.403574 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.403590 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.403594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.403653 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.403878 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.403916 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:40:11.404155 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.398558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33687 (local address 127.28.62.130:35643)
0809 16:40:11.398766 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:11.398810 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:11.399390 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:11.399700 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:11.399704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:11.399712 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:11.400247 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.400251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.401071 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.401072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.402096 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.402100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.402555 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.402556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.402730 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.403396 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:11.403397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:11.403437 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:11.403468 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:11.403665 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:11.403672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:11.403675 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:40:11.403749 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:40:11.403877 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260809 16:40:11.404939 7568 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33687
I20260809 16:40:11.405059 7568 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:11.405238 7568 heartbeater.cc:511] Master 127.28.62.190:33687 requested a full tablet report, sending...
I20260809 16:40:11.405683 7193 ts_manager.cc:194] Registered new tserver with Master: 5c9f36628a2846adab33ed568c95244a (127.28.62.130:42383)
I20260809 16:40:11.406201 7193 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.130:35643
W20260809 16:40:11.488299 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.488562 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.488585 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.489905 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.489954 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.489982 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:40:11.491408 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.491677 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.492024 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.497480 7578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.497773 7582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.497753 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.497844 7571 server_base.cc:1034] running on GCE node
I20260809 16:40:11.498097 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.498327 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.499475 7571 hybrid_clock.cc:648] HybridClock initialized: now 1786293611499440 us; error 52 us; skew 500 ppm
I20260809 16:40:11.500898 7571 webserver.cc:460] Webserver started at http://127.28.62.131:38113/ using document root <none> and password file <none>
I20260809 16:40:11.501187 7571 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.501263 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.501425 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.502557 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "c9c2accc0a3844bca9a1470492b4d5ed"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7665a4ff8dc0c4e3b626ca60da17113d"
server_key_iv: "81d5aefc9cc83cfefa43c792be26ad38"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.502947 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c2accc0a3844bca9a1470492b4d5ed"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7665a4ff8dc0c4e3b626ca60da17113d"
server_key_iv: "81d5aefc9cc83cfefa43c792be26ad38"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.504427 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:11.505216 7589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.505416 7571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:11.505523 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "c9c2accc0a3844bca9a1470492b4d5ed"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "7665a4ff8dc0c4e3b626ca60da17113d"
server_key_iv: "81d5aefc9cc83cfefa43c792be26ad38"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.505602 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.535317 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.535756 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.535928 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.536213 7571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:11.536589 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:11.536651 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.536688 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:11.536721 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.542682 7571 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44939
I20260809 16:40:11.542762 7724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44939 every 8 connection(s)
I20260809 16:40:11.543140 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:40:11.548527 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.543930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:46487 (local address 127.28.62.190:33687)
0809 16:40:11.544004 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.544006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.544229 (+ 223us) server_negotiation.cc:407] Connection header received
0809 16:40:11.544748 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.544749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.544759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.544837 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.545481 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.545993 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.547108 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.547342 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.547429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.548266 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.548268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.548268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.548273 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.548286 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.548291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.548345 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.548427 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.548463 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:40:11.548764 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.543901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33687 (local address 127.28.62.131:46487)
0809 16:40:11.544097 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:11.544139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:11.544663 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:11.544892 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:11.544895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:11.544903 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:11.545401 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.545404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.546115 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.546116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.546951 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:11.546954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:11.547450 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:11.547451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:11.547568 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:11.548108 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:11.548108 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:11.548137 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:11.548160 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:11.548357 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:11.548362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:11.548364 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:40:11.548416 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:40:11.548500 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260809 16:40:11.549408 7725 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33687
I20260809 16:40:11.549430 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7571
I20260809 16:40:11.549522 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:40:11.549535 7725 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:11.549751 7725 heartbeater.cc:511] Master 127.28.62.190:33687 requested a full tablet report, sending...
I20260809 16:40:11.549765 28922 external_mini_cluster.cc:1428] Setting key 5c4f8ed5a7eaeec99c0ce04af03d3b17
I20260809 16:40:11.550199 7192 ts_manager.cc:194] Registered new tserver with Master: c9c2accc0a3844bca9a1470492b4d5ed (127.28.62.131:44939)
I20260809 16:40:11.550680 7192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.131:46487
I20260809 16:40:11.550887 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 16:40:11.552738 7257 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:11.552250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.28.62.190:33687)
0809 16:40:11.552332 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.552334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.552440 (+ 106us) server_negotiation.cc:407] Connection header received
0809 16:40:11.552477 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.552478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.552489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.552536 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.552677 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47280: BlockingRecv error: recv got EOF from 127.0.0.1:47280 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 16:40:11.552771 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7264
I20260809 16:40:11.567582 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7416
I20260809 16:40:11.574504 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7571
I20260809 16:40:11.580488 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7160
2026-08-09T16:40:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (655 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T16:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:40:11Z Disabled control of system clock
I20260809 16:40:11.610643 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:33313
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 16:40:11.697482 7739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.697718 7739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.697743 7739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.698940 7739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:40:11.698977 7739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.698989 7739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.699002 7739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:40:11.699087 7739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:40:11.700464 7739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:33313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.700675 7739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.700941 7739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.705962 7749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.705988 7751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.705988 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.706308 7739 server_base.cc:1034] running on GCE node
I20260809 16:40:11.706593 7739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.706918 7739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.708097 7739 hybrid_clock.cc:648] HybridClock initialized: now 1786293611708089 us; error 54 us; skew 500 ppm
I20260809 16:40:11.709465 7739 webserver.cc:460] Webserver started at http://127.28.62.190:35989/ using document root <none> and password file <none>
I20260809 16:40:11.709734 7739 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.709810 7739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.709962 7739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.711124 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "7a6550596c1648e1b13f42df3a84dcd0"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "aab6ca7eb2a954313f20dc35fa7ea8f2"
server_key_iv: "d528a69e59f1b60c282157f8bac398bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.711490 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "7a6550596c1648e1b13f42df3a84dcd0"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "aab6ca7eb2a954313f20dc35fa7ea8f2"
server_key_iv: "d528a69e59f1b60c282157f8bac398bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.712877 7739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:40:11.713660 7757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.713843 7739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:40:11.713946 7739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "7a6550596c1648e1b13f42df3a84dcd0"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "aab6ca7eb2a954313f20dc35fa7ea8f2"
server_key_iv: "d528a69e59f1b60c282157f8bac398bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.714030 7739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.717553 7739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.717833 7739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.717978 7739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.722050 7739 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:33313
I20260809 16:40:11.722083 7822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:33313 every 8 connection(s)
I20260809 16:40:11.722422 7739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:40:11.723136 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:11.725870 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0: Bootstrap starting.
I20260809 16:40:11.725999 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7739
I20260809 16:40:11.726101 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:40:11.726300 28922 external_mini_cluster.cc:1428] Setting key 809ce05498837e1b150af61fd05482d8
I20260809 16:40:11.726578 7823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:11.726963 7823 log.cc:824] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:11.727712 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0: No bootstrap required, opened a new log
I20260809 16:40:11.728924 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.726871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.28.62.190:33313)
0809 16:40:11.727130 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.727142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.727168 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:40:11.727592 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.727598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.727635 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.727786 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.728494 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.728501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.728502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.728506 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.728542 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.728552 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.728648 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.728706 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.728768 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260809 16:40:11.729707 7823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } }
I20260809 16:40:11.729877 7823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:11.729928 7823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6550596c1648e1b13f42df3a84dcd0, State: Initialized, Role: FOLLOWER
I20260809 16:40:11.730046 7823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } }
I20260809 16:40:11.730140 7823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:40:11.730185 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:40:11.730239 7823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:11.731168 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } }
I20260809 16:40:11.731292 7823 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6550596c1648e1b13f42df3a84dcd0; no voters:
I20260809 16:40:11.731459 7823 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:40:11.731560 7830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:40:11.731760 7830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a6550596c1648e1b13f42df3a84dcd0, State: Running, Role: LEADER
I20260809 16:40:11.731819 7823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:40:11.731941 7830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } }
I20260809 16:40:11.732452 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6550596c1648e1b13f42df3a84dcd0. Latest consensus state: current_term: 1 leader_uuid: "7a6550596c1648e1b13f42df3a84dcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } } }
I20260809 16:40:11.732734 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [sys.catalog]: This master's current role is: LEADER
I20260809 16:40:11.732440 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a6550596c1648e1b13f42df3a84dcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6550596c1648e1b13f42df3a84dcd0" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 33313 } } }
I20260809 16:40:11.732930 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0 [sys.catalog]: This master's current role is: LEADER
I20260809 16:40:11.732964 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:40:11.733498 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:40:11.735308 7842 catalog_manager.cc:1347] Generated new cluster ID: 51c423f227964a73951fc0bd3a7ee1ad
I20260809 16:40:11.735368 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:40:11.746661 7842 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:40:11.746876 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:40:11.748754 7842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a6550596c1648e1b13f42df3a84dcd0: Generated new TSK 0
I20260809 16:40:11.748960 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:40:11.755810 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--builtin_ntp_servers=127.28.62.148:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 16:40:11.864835 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.865097 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.865139 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.866351 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.866421 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.866468 7854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:40:11.867992 7854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.868283 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.868607 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.874832 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.874866 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.875138 7863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.875195 7854 server_base.cc:1034] running on GCE node
I20260809 16:40:11.875577 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.875896 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.877091 7854 hybrid_clock.cc:648] HybridClock initialized: now 1786293611877077 us; error 30 us; skew 500 ppm
I20260809 16:40:11.878810 7854 webserver.cc:460] Webserver started at http://127.28.62.129:32769/ using document root <none> and password file <none>
I20260809 16:40:11.879138 7854 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:11.879213 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:11.879362 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:11.880844 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "a65843af3a664023aa2bdc570581f8f4"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "df1fde86a1d7b7cdc01b9ea103503925"
server_key_iv: "0f34c9ca5f08664314c8d2aec09f31eb"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.881245 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "a65843af3a664023aa2bdc570581f8f4"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "df1fde86a1d7b7cdc01b9ea103503925"
server_key_iv: "0f34c9ca5f08664314c8d2aec09f31eb"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.882915 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:40:11.883895 7869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.884101 7854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:11.884197 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "a65843af3a664023aa2bdc570581f8f4"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "df1fde86a1d7b7cdc01b9ea103503925"
server_key_iv: "0f34c9ca5f08664314c8d2aec09f31eb"
server_key_version: "encryptionkey@0"
I20260809 16:40:11.884274 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:11.889052 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:11.889365 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:11.889523 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:11.889779 7854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:11.890153 7854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:11.890208 7854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.890236 7854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:11.890254 7854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:11.897809 7854 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:46005
I20260809 16:40:11.897846 8003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:46005 every 8 connection(s)
I20260809 16:40:11.898222 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:40:11.901580 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.899137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:55587 (local address 127.28.62.190:33313)
0809 16:40:11.899236 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:11.899238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:11.899426 (+ 188us) server_negotiation.cc:407] Connection header received
0809 16:40:11.899967 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:11.899968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:11.899974 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:11.900032 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:11.901293 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:11.901294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:11.901295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:11.901300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:11.901316 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:11.901320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:11.901390 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:11.901485 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 16:40:11.901502 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:40:11.901858 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:11.899093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33313 (local address 127.28.62.129:55587)
0809 16:40:11.899297 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:11.899332 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:11.899855 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:11.900106 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:11.900116 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:11.900125 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:11.901117 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:11.901118 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:11.901156 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:11.901186 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:11.901387 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:11.901394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:11.901397 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:40:11.901495 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 16:40:11.901557 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260809 16:40:11.902400 8004 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33313
I20260809 16:40:11.902505 8004 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:11.902667 8004 heartbeater.cc:511] Master 127.28.62.190:33313 requested a full tablet report, sending...
I20260809 16:40:11.903028 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 7854
I20260809 16:40:11.903124 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:40:11.903258 7776 ts_manager.cc:194] Registered new tserver with Master: a65843af3a664023aa2bdc570581f8f4 (127.28.62.129:46005)
I20260809 16:40:11.903322 28922 external_mini_cluster.cc:1428] Setting key f535f4ac8bfd9de7ea31b48b297a130f
I20260809 16:40:11.904312 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--builtin_ntp_servers=127.28.62.148:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 16:40:11.987932 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:11.988253 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:11.988296 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:11.989642 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:11.989711 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:11.989758 8008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:40:11.991194 8008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:11.991472 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:11.991796 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:11.996994 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.997001 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:11.997008 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:11.997205 8008 server_base.cc:1034] running on GCE node
I20260809 16:40:11.997520 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:11.997785 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:11.998927 8008 hybrid_clock.cc:648] HybridClock initialized: now 1786293611998914 us; error 28 us; skew 500 ppm
I20260809 16:40:12.000346 8008 webserver.cc:460] Webserver started at http://127.28.62.130:39429/ using document root <none> and password file <none>
I20260809 16:40:12.000626 8008 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:12.000675 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:12.000839 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:12.001976 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "18588a287f2c40d4b4ad557bc5a2a630"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "4c82a0a5158356be8045f36edf451634"
server_key_iv: "6904f858509c0332f5b85dc670c60a8b"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.002348 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "18588a287f2c40d4b4ad557bc5a2a630"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "4c82a0a5158356be8045f36edf451634"
server_key_iv: "6904f858509c0332f5b85dc670c60a8b"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.003909 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:40:12.004777 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.004958 8008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:12.005064 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "18588a287f2c40d4b4ad557bc5a2a630"
format_stamp: "Formatted at 2026-08-09 16:40:11 on dist-test-slave-wl2h"
server_key: "4c82a0a5158356be8045f36edf451634"
server_key_iv: "6904f858509c0332f5b85dc670c60a8b"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.005148 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:12.008814 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:12.009088 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:12.009240 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:12.009456 8008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:12.009776 8008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:12.009830 8008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.009864 8008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:12.009887 8008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.016224 8008 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:33689
I20260809 16:40:12.016321 8159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:33689 every 8 connection(s)
I20260809 16:40:12.016673 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:40:12.020215 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8008
I20260809 16:40:12.020344 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:40:12.020424 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:12.017662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60927 (local address 127.28.62.190:33313)
0809 16:40:12.017708 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:12.017711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:12.017994 (+ 283us) server_negotiation.cc:407] Connection header received
0809 16:40:12.018557 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:12.018559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:12.018565 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:12.018640 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:12.019973 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:12.019974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:12.019974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:12.019978 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:12.019996 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:12.020000 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:12.020077 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:12.020321 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 16:40:12.020342 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:40:12.020546 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:12.017627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33313 (local address 127.28.62.130:60927)
0809 16:40:12.017833 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:12.017876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:12.018429 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:12.018716 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:12.018720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:12.018728 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:12.019756 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:12.019757 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:12.019787 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:12.019817 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:12.020091 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:12.020097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:12.020101 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:40:12.020210 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 16:40:12.020292 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
I20260809 16:40:12.020557 28922 external_mini_cluster.cc:1428] Setting key 66a88a8f3fa97c94aa6fd944f56f3c1e
I20260809 16:40:12.022003 8160 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33313
I20260809 16:40:12.022132 8160 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:12.022337 8160 heartbeater.cc:511] Master 127.28.62.190:33313 requested a full tablet report, sending...
I20260809 16:40:12.022333 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--builtin_ntp_servers=127.28.62.148:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 16:40:12.022856 7776 ts_manager.cc:194] Registered new tserver with Master: 18588a287f2c40d4b4ad557bc5a2a630 (127.28.62.130:33689)
W20260809 16:40:12.105432 8165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:12.105644 8165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:12.105669 8165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:12.106877 8165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:12.106920 8165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:12.106946 8165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:40:12.108487 8165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:12.108747 8165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:12.109095 8165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:12.114108 8170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:12.114118 8175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:12.114275 8165 server_base.cc:1034] running on GCE node
W20260809 16:40:12.114118 8172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:12.114598 8165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:12.114822 8165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:12.115971 8165 hybrid_clock.cc:648] HybridClock initialized: now 1786293612115928 us; error 60 us; skew 500 ppm
I20260809 16:40:12.117453 8165 webserver.cc:460] Webserver started at http://127.28.62.131:46103/ using document root <none> and password file <none>
I20260809 16:40:12.117740 8165 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:12.117796 8165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:12.118003 8165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:12.119323 8165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "5e17b821eaeb4d37a9178627c22268ae"
format_stamp: "Formatted at 2026-08-09 16:40:12 on dist-test-slave-wl2h"
server_key: "ba79a1c716ddc1390a380f367653118e"
server_key_iv: "03b7aa097419d5a72fa8190a3bbeefe5"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.119719 8165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "5e17b821eaeb4d37a9178627c22268ae"
format_stamp: "Formatted at 2026-08-09 16:40:12 on dist-test-slave-wl2h"
server_key: "ba79a1c716ddc1390a380f367653118e"
server_key_iv: "03b7aa097419d5a72fa8190a3bbeefe5"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.121445 8165 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 16:40:12.122392 8182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.122709 8165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:12.122818 8165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "5e17b821eaeb4d37a9178627c22268ae"
format_stamp: "Formatted at 2026-08-09 16:40:12 on dist-test-slave-wl2h"
server_key: "ba79a1c716ddc1390a380f367653118e"
server_key_iv: "03b7aa097419d5a72fa8190a3bbeefe5"
server_key_version: "encryptionkey@0"
I20260809 16:40:12.122901 8165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:12.126740 8165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:12.127091 8165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:12.127276 8165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:12.127521 8165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:12.127882 8165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:12.127928 8165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.127975 8165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:12.128007 8165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:12.134809 8165 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44201
I20260809 16:40:12.134851 8311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44201 every 8 connection(s)
I20260809 16:40:12.135308 8165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:40:12.138424 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8165
I20260809 16:40:12.138538 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:40:12.138697 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:12.136169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:49407 (local address 127.28.62.190:33313)
0809 16:40:12.136238 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:12.136242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:12.136481 (+ 239us) server_negotiation.cc:407] Connection header received
0809 16:40:12.136966 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:12.136967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:12.136971 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:12.137034 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:12.138425 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:12.138426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:12.138427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:40:12.138429 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:12.138442 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:40:12.138445 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:12.138505 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:12.138620 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 16:40:12.138643 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 16:40:12.138799 28922 external_mini_cluster.cc:1428] Setting key 90538bed3cf7eb132012251c5c793ba4
I20260809 16:40:12.138950 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:12.136135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:33313 (local address 127.28.62.131:49407)
0809 16:40:12.136343 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:12.136388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:12.136878 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:12.137089 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:12.137092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:12.137101 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:12.138224 (+ 1123us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:40:12.138225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:12.138269 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:40:12.138312 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:12.138489 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:12.138494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:12.138498 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:40:12.138628 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 16:40:12.138703 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260809 16:40:12.139515 8312 heartbeater.cc:347] Connected to a master server at 127.28.62.190:33313
I20260809 16:40:12.139616 8312 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:12.139779 8312 heartbeater.cc:511] Master 127.28.62.190:33313 requested a full tablet report, sending...
I20260809 16:40:12.140178 7776 ts_manager.cc:194] Registered new tserver with Master: 5e17b821eaeb4d37a9178627c22268ae (127.28.62.131:44201)
I20260809 16:40:12.150939 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 16:40:12.152911 7827 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:40:12.152492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.28.62.190:33313)
0809 16:40:12.152537 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:12.152539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:12.152608 (+ 69us) server_negotiation.cc:407] Connection header received
0809 16:40:12.152632 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:12.152632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:12.152637 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:12.152678 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:12.152833 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46306: BlockingRecv error: recv got EOF from 127.0.0.1:46306 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:40:12.152994 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7854
I20260809 16:40:12.159492 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8008
I20260809 16:40:12.166152 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8165
I20260809 16:40:12.171829 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 7739
2026-08-09T16:40:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (592 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:40:12 dist-test-slave-wl2h krb5kdc[8322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:40:12 dist-test-slave-wl2h krb5kdc[8322](info): set up 2 sockets
Aug 09 16:40:12 dist-test-slave-wl2h krb5kdc[8322](info): commencing operation
krb5kdc: starting...
W20260809 16:40:13.309926 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:40:13.447383 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35809
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:40:13.533680 8337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:13.533958 8337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:13.534001 8337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:13.535343 8337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:40:13.535422 8337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:13.535450 8337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:13.535488 8337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:40:13.535519 8337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:40:13.537168 8337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:13.537482 8337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:13.537740 8337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:13.543182 8345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.543169 8343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.543160 8342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:13.543629 8337 server_base.cc:1034] running on GCE node
I20260809 16:40:13.543881 8337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:13.544175 8337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:13.545348 8337 hybrid_clock.cc:648] HybridClock initialized: now 1786293613545333 us; error 30 us; skew 500 ppm
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:40:13.548002 8337 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:40:13.548548 8337 webserver.cc:460] Webserver started at http://127.28.62.190:39705/ using document root <none> and password file <none>
I20260809 16:40:13.548792 8337 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:13.548862 8337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:13.549018 8337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:13.550277 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "e0de2d52598a4879810d073506f75c6f"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "ccb0c395d0f408099599edec2bb07ca3"
server_key_iv: "b450e26e77b85ca25e5a73e8a4328463"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.550663 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "e0de2d52598a4879810d073506f75c6f"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "ccb0c395d0f408099599edec2bb07ca3"
server_key_iv: "b450e26e77b85ca25e5a73e8a4328463"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.552246 8337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:40:13.553164 8352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.553385 8337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:13.553498 8337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "e0de2d52598a4879810d073506f75c6f"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "ccb0c395d0f408099599edec2bb07ca3"
server_key_iv: "b450e26e77b85ca25e5a73e8a4328463"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.553575 8337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:13.581414 8337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:13.583940 8337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:13.584122 8337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:13.588559 8337 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35809
I20260809 16:40:13.588617 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35809 every 8 connection(s)
I20260809 16:40:13.589006 8337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:40:13.590018 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:13.592465 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: Bootstrap starting.
I20260809 16:40:13.593101 8413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:13.593420 8413 log.cc:824] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:13.593832 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8337
I20260809 16:40:13.593964 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:40:13.594183 28922 external_mini_cluster.cc:1428] Setting key e69ae9bffade2223bfb3c7c6019a5689
I20260809 16:40:13.594380 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: No bootstrap required, opened a new log
I20260809 16:40:13.596375 8413 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } }
I20260809 16:40:13.596515 8413 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:13.596537 8413 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0de2d52598a4879810d073506f75c6f, State: Initialized, Role: FOLLOWER
I20260809 16:40:13.596763 8413 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } }
I20260809 16:40:13.596864 8413 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:40:13.596902 8413 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:40:13.596949 8413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:13.598050 8413 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } }
I20260809 16:40:13.598147 8413 leader_election.cc:302] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0de2d52598a4879810d073506f75c6f; no voters:
I20260809 16:40:13.598357 8413 leader_election.cc:288] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:40:13.598407 8418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:40:13.598562 8418 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [term 1 LEADER]: Becoming Leader. State: Replica: e0de2d52598a4879810d073506f75c6f, State: Running, Role: LEADER
I20260809 16:40:13.598649 8413 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:40:13.598712 8418 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } }
I20260809 16:40:13.599220 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0de2d52598a4879810d073506f75c6f. Latest consensus state: current_term: 1 leader_uuid: "e0de2d52598a4879810d073506f75c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } } }
I20260809 16:40:13.599370 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [sys.catalog]: This master's current role is: LEADER
I20260809 16:40:13.599896 8429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:40:13.600401 8429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:40:13.600406 8419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0de2d52598a4879810d073506f75c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0de2d52598a4879810d073506f75c6f" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35809 } } }
I20260809 16:40:13.600718 8419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f [sys.catalog]: This master's current role is: LEADER
I20260809 16:40:13.602264 8429 catalog_manager.cc:1347] Generated new cluster ID: da082c0650da4666808d1da877d11d49
I20260809 16:40:13.602319 8429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:40:13.607263 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:13.594780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.28.62.190:35809)
0809 16:40:13.594989 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:13.594998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:13.595062 (+ 64us) server_negotiation.cc:407] Connection header received
0809 16:40:13.596369 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:13.596377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:13.596443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:13.596616 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:13.597478 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.598666 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.599837 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.600521 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.600645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:13.603374 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:13.603378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:13.603378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:40:13.603390 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:13.605494 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.605889 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.605893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.605893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.606037 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.606253 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.606254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.606255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.606579 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:13.606754 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:13.606834 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:40:13.606988 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260809 16:40:13.615702 8429 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:40:13.616430 8429 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:40:13.623183 8429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: Generated new TSK 0
I20260809 16:40:13.623382 8429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:40:13.655479 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:40:13.743530 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:13.743783 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:13.743825 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:13.745057 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:13.745137 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:13.745184 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:40:13.746687 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:13.746979 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:13.747269 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:13.752671 8456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.752655 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.752660 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:13.752970 8445 server_base.cc:1034] running on GCE node
I20260809 16:40:13.753266 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:13.753541 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:13.754724 8445 hybrid_clock.cc:648] HybridClock initialized: now 1786293613754695 us; error 39 us; skew 500 ppm
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:40:13.762708 8445 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:40:13.763242 8445 webserver.cc:460] Webserver started at http://127.28.62.129:37773/ using document root <none> and password file <none>
I20260809 16:40:13.763499 8445 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:13.763571 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:13.763724 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:13.764819 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "ebfb02d1b9da4c34b00ab99348267813"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "06230db7a2a0c09fee276ecddfa4c450"
server_key_iv: "5441050f74220c56f7b7fbf49b4144d2"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.765197 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "ebfb02d1b9da4c34b00ab99348267813"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "06230db7a2a0c09fee276ecddfa4c450"
server_key_iv: "5441050f74220c56f7b7fbf49b4144d2"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.766563 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:40:13.767395 8464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.767696 8445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:13.767805 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "ebfb02d1b9da4c34b00ab99348267813"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "06230db7a2a0c09fee276ecddfa4c450"
server_key_iv: "5441050f74220c56f7b7fbf49b4144d2"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.767882 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:13.790510 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:13.793017 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:13.793207 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:13.793466 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:13.793848 8445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:13.793905 8445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.793951 8445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:13.793993 8445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.800345 8445 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:42209
I20260809 16:40:13.800495 8593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:42209 every 8 connection(s)
I20260809 16:40:13.800783 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:40:13.802276 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8445
I20260809 16:40:13.802397 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:40:13.802593 28922 external_mini_cluster.cc:1428] Setting key 2c09279d888aeab5c40d44e7f58eee7a
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:40:13.811414 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:13.802015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47425 (local address 127.28.62.190:35809)
0809 16:40:13.802070 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:13.802072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:13.802299 (+ 227us) server_negotiation.cc:407] Connection header received
0809 16:40:13.802982 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:13.802984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:13.803001 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:13.803085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:13.804200 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.804728 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.805762 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.805996 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.806076 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:13.808392 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:13.808394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:13.808395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:40:13.808403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:13.809882 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.810327 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.810328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.810328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.810388 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.810646 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.810647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.810647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.810775 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:13.810850 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:13.811222 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 16:40:13.811321 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:40:13.811602 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:13.801962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35809 (local address 127.28.62.129:47425)
0809 16:40:13.802167 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:13.802204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:13.802886 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:13.803167 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:13.803172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:13.803866 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:13.804077 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:13.804082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.804838 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.804839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:13.805656 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:13.805658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.806089 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.806090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:13.806237 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:13.806800 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:13.806808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:13.808287 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:13.809992 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:13.809999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:13.810002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:13.810236 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:13.810458 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:13.810459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:13.810459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:13.810574 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:13.810853 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:13.810861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:13.811002 (+ 141us) client_negotiation.cc:769] Sending connection context
0809 16:40:13.811195 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 16:40:13.811362 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":117,"threads_started":1}
I20260809 16:40:13.812250 8594 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35809
I20260809 16:40:13.812368 8594 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:13.812552 8594 heartbeater.cc:511] Master 127.28.62.190:35809 requested a full tablet report, sending...
I20260809 16:40:13.813143 8374 ts_manager.cc:194] Registered new tserver with Master: ebfb02d1b9da4c34b00ab99348267813 (127.28.62.129:42209)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
I20260809 16:40:13.813917 8374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:47425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:40:13.829892 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:40:13.912009 8602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:13.912238 8602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:13.912261 8602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:13.913321 8602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:13.913357 8602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:13.913383 8602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:40:13.914768 8602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:13.914954 8602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:13.915207 8602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:13.920328 8608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.920342 8609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:13.920337 8611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:13.920550 8602 server_base.cc:1034] running on GCE node
I20260809 16:40:13.920904 8602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:13.921108 8602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:13.922255 8602 hybrid_clock.cc:648] HybridClock initialized: now 1786293613922231 us; error 39 us; skew 500 ppm
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:40:13.924875 8602 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:40:13.925333 8602 webserver.cc:460] Webserver started at http://127.28.62.130:39129/ using document root <none> and password file <none>
I20260809 16:40:13.925575 8602 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:13.925621 8602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:13.925796 8602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:13.926854 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "fedfba342ff24249aefa978cc2a61dfb"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "7aae8c8cec819b5b60366bd2486c69db"
server_key_iv: "591f0b3b890b46543f442834790d69ef"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.927243 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "fedfba342ff24249aefa978cc2a61dfb"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "7aae8c8cec819b5b60366bd2486c69db"
server_key_iv: "591f0b3b890b46543f442834790d69ef"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.928570 8602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:13.929428 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.929612 8602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:40:13.929728 8602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "fedfba342ff24249aefa978cc2a61dfb"
format_stamp: "Formatted at 2026-08-09 16:40:13 on dist-test-slave-wl2h"
server_key: "7aae8c8cec819b5b60366bd2486c69db"
server_key_iv: "591f0b3b890b46543f442834790d69ef"
server_key_version: "encryptionkey@0"
I20260809 16:40:13.929803 8602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:13.939765 8602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:13.941650 8602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:13.941821 8602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:13.942052 8602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:13.942364 8602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:13.942420 8602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.942463 8602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:13.942492 8602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:13.948639 8602 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:37013
I20260809 16:40:13.948686 8756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:37013 every 8 connection(s)
I20260809 16:40:13.949239 8602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:40:13 dist-test-slave-wl2h krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293613, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:40:13.955607 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8602
I20260809 16:40:13.955713 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:40:13.955977 28922 external_mini_cluster.cc:1428] Setting key 5084a6a6c6abb1714a1c41f8624643f1
I20260809 16:40:13.959215 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:13.950040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:37953 (local address 127.28.62.190:35809)
0809 16:40:13.950089 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:13.950091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:13.950273 (+ 182us) server_negotiation.cc:407] Connection header received
0809 16:40:13.950783 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:13.950784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:13.950794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:13.950843 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:13.951804 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.952386 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.953328 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.953544 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.953601 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:13.955906 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:13.955908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:13.955908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:40:13.955916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:13.957589 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.958087 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.958088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.958088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.958145 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:13.958441 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:13.958441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:13.958442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:13.958551 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:13.958650 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:13.958977 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 16:40:13.959120 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:40:13.959441 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:13.949999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35809 (local address 127.28.62.130:37953)
0809 16:40:13.950157 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:13.950198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:13.950668 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:13.950892 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:13.950895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:13.951523 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:13.951720 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:13.951725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.952474 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.952474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:13.953248 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:13.953250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:13.953622 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:13.953622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:13.953732 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:13.954222 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:13.954231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:13.955789 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:13.957671 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:13.957679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:13.957681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:13.958006 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:13.958215 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:13.958216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:13.958216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:13.958371 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:13.958645 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:13.958654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:13.958817 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 16:40:13.958939 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 16:40:13.959147 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260809 16:40:13.960047 8758 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35809
I20260809 16:40:13.960147 8758 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:13.960366 8758 heartbeater.cc:511] Master 127.28.62.190:35809 requested a full tablet report, sending...
I20260809 16:40:13.960789 8374 ts_manager.cc:194] Registered new tserver with Master: fedfba342ff24249aefa978cc2a61dfb (127.28.62.130:37013)
I20260809 16:40:13.961337 8374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:37953
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:40:13.987792 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:40:14.070958 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:40:14.071319 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:40:14.071446 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:40:14.072656 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:40:14.072726 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:40:14.072775 8763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:40:14.074221 8763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:40:14.074489 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:40:14.074733 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:40:14.079926 8770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:14.079950 8772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:40:14.079932 8769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:40:14.080307 8763 server_base.cc:1034] running on GCE node
I20260809 16:40:14.080514 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:40:14.080765 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:40:14.081939 8763 hybrid_clock.cc:648] HybridClock initialized: now 1786293614081907 us; error 44 us; skew 500 ppm
Aug 09 16:40:14 dist-test-slave-wl2h krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293614, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:40:14.084656 8763 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:40:14.085180 8763 webserver.cc:460] Webserver started at http://127.28.62.131:46563/ using document root <none> and password file <none>
I20260809 16:40:14.085402 8763 fs_manager.cc:359] Metadata directory not provided
I20260809 16:40:14.085444 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:40:14.085533 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:40:14.086525 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "cbe7e088381c4c018224a8165ad2c493"
format_stamp: "Formatted at 2026-08-09 16:40:14 on dist-test-slave-wl2h"
server_key: "9aa629b4edc75a5832f453840b8a94c5"
server_key_iv: "7f5d04ed17e5cd62c219500a1f1523bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:14.086849 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "cbe7e088381c4c018224a8165ad2c493"
format_stamp: "Formatted at 2026-08-09 16:40:14 on dist-test-slave-wl2h"
server_key: "9aa629b4edc75a5832f453840b8a94c5"
server_key_iv: "7f5d04ed17e5cd62c219500a1f1523bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:14.088239 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:40:14.089025 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:14.089294 8763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:40:14.089392 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "cbe7e088381c4c018224a8165ad2c493"
format_stamp: "Formatted at 2026-08-09 16:40:14 on dist-test-slave-wl2h"
server_key: "9aa629b4edc75a5832f453840b8a94c5"
server_key_iv: "7f5d04ed17e5cd62c219500a1f1523bf"
server_key_version: "encryptionkey@0"
I20260809 16:40:14.089453 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:40:14.117978 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:40:14.120723 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:40:14.120852 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:40:14.121078 8763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:40:14.121443 8763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:40:14.121482 8763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:14.121507 8763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:40:14.121567 8763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:40:14.128317 8763 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36579
I20260809 16:40:14.128366 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36579 every 8 connection(s)
I20260809 16:40:14.128728 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:40:14.133760 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 8763
I20260809 16:40:14.133983 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:40:14.134248 28922 external_mini_cluster.cc:1428] Setting key b08c039ec7ed707218de79ae21a0beef
Aug 09 16:40:14 dist-test-slave-wl2h krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293614, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:40:14.138607 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.129521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:44975 (local address 127.28.62.190:35809)
0809 16:40:14.129594 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.129596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.129771 (+ 175us) server_negotiation.cc:407] Connection header received
0809 16:40:14.130326 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.130328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.130337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.130386 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:14.131460 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.131993 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.133142 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.133327 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.133383 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.136133 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:14.136135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:14.136135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:40:14.136141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:14.137378 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:14.137782 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:14.137783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:14.137783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:14.137832 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:14.138054 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:14.138054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:14.138054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:14.138149 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:14.138199 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.138478 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.138544 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:40:14.138854 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.129462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35809 (local address 127.28.62.131:44975)
0809 16:40:14.129639 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.129679 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.130235 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.130447 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.130450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.131159 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:40:14.131374 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.131379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.132139 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.132140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.133045 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.133047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.133408 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.133409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.133508 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.134193 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:40:14.134203 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:40:14.136042 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:40:14.137454 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:14.137461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:14.137463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:14.137713 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:14.137894 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:40:14.137895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:40:14.137895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:40:14.137998 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:40:14.138234 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:40:14.138239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:40:14.138359 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.138454 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.138579 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260809 16:40:14.139364 8908 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35809
I20260809 16:40:14.139477 8908 heartbeater.cc:464] Registering TS with master...
I20260809 16:40:14.139675 8908 heartbeater.cc:511] Master 127.28.62.190:35809 requested a full tablet report, sending...
I20260809 16:40:14.140081 8374 ts_manager.cc:194] Registered new tserver with Master: cbe7e088381c4c018224a8165ad2c493 (127.28.62.131:36579)
I20260809 16:40:14.140462 8374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:44975
I20260809 16:40:14.146724 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:40:14.154351 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.148162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.28.62.190:35809)
0809 16:40:14.148262 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.148264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.148390 (+ 126us) server_negotiation.cc:407] Connection header received
0809 16:40:14.148469 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.148470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.148481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.148526 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:40:14.149097 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.149599 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.150326 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.150521 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.150596 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.151851 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:40:14.151852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:40:14.151853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:40:14.151860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:40:14.153151 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:14.153501 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:14.153502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:14.153503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:14.153566 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:40:14.153776 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:40:14.153777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:40:14.153777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:40:14.153945 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:40:14.154058 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.154160 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.154247 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 16:40:14.156147 8374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:40:14.157817 8374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:40:14.164750 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.28.62.131:36579)
0809 16:40:14.160873 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.160878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.160963 (+ 85us) server_negotiation.cc:407] Connection header received
0809 16:40:14.161047 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.161049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.161068 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.161153 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:40:14.161437 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162576 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.163812 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164345 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.164436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.164494 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.164623 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.164664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260809 16:40:14.164800 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36579 (local address 127.0.0.1:57652)
0809 16:40:14.160881 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.160890 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.160969 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.161243 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.161246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.161249 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:40:14.161318 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.161338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162711 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.162712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.163717 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.163720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164435 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.164435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.164514 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.164527 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.164598 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.164653 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":12,"thread_start_us":75,"threads_started":1}
I20260809 16:40:14.164816 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42209 (local address 127.0.0.1:56826)
0809 16:40:14.160930 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.160940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.160995 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.161390 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.161391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.161393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:40:14.161436 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.161437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162299 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.162300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.163510 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.163513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164533 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.164534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.164599 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.164603 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.164658 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.164713 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260809 16:40:14.164927 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:37013 (local address 127.0.0.1:55570)
0809 16:40:14.160453 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.160484 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.160567 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.161283 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.161284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.161285 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:40:14.161317 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.161319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162690 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.162691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.163838 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.163841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164736 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.164737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.164796 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.164799 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.164828 (+ 29us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.164867 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":32,"thread_start_us":71,"threads_started":1}
I20260809 16:40:14.165056 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.28.62.129:42209)
0809 16:40:14.161103 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.161108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.161118 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:40:14.161164 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.161166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.161183 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.161304 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:40:14.161519 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162198 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.163602 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164367 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.164438 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.164495 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.164914 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.164968 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260809 16:40:14.165572 8830 tablet_service.cc:1467] Processing CreateTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af (DEFAULT_TABLE table=test-table [id=6aaa51e19a9b40a0ba4d8589007d65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:40:14.165834 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.160438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.28.62.130:37013)
0809 16:40:14.161103 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.161108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.161117 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:40:14.161170 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.161171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.161185 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.161395 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:40:14.161473 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.162576 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.163923 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.164635 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.165606 (+ 971us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.165682 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.165710 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.165758 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":108,"threads_started":1}
I20260809 16:40:14.165748 8519 tablet_service.cc:1467] Processing CreateTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af (DEFAULT_TABLE table=test-table [id=6aaa51e19a9b40a0ba4d8589007d65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:40:14.165990 8830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f618145bb0e4450bccb625ba3f9e9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:14.166090 8519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f618145bb0e4450bccb625ba3f9e9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:14.168893 8932 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Bootstrap starting.
I20260809 16:40:14.169406 8679 tablet_service.cc:1467] Processing CreateTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af (DEFAULT_TABLE table=test-table [id=6aaa51e19a9b40a0ba4d8589007d65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:40:14.169685 8932 tablet_bootstrap.cc:654] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:14.169768 8679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f618145bb0e4450bccb625ba3f9e9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:40:14.170313 8932 log.cc:824] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:14.170725 8930 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Bootstrap starting.
I20260809 16:40:14.171311 8932 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: No bootstrap required, opened a new log
I20260809 16:40:14.171411 8932 ts_tablet_manager.cc:1397] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:40:14.171448 8930 tablet_bootstrap.cc:654] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:14.172133 8930 log.cc:824] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:14.173012 8930 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: No bootstrap required, opened a new log
I20260809 16:40:14.173087 8930 ts_tablet_manager.cc:1397] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:40:14.173404 8935 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Bootstrap starting.
I20260809 16:40:14.173673 8932 raft_consensus.cc:348] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.173889 8932 raft_consensus.cc:374] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:14.173930 8932 raft_consensus.cc:729] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe7e088381c4c018224a8165ad2c493, State: Initialized, Role: FOLLOWER
I20260809 16:40:14.174080 8932 consensus_queue.cc:260] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.174193 8935 tablet_bootstrap.cc:654] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Neither blocks nor log segments found. Creating new log.
I20260809 16:40:14.174433 8908 heartbeater.cc:503] Master 127.28.62.190:35809 was elected leader, sending a full tablet report...
I20260809 16:40:14.174448 8932 ts_tablet_manager.cc:1428] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:40:14.174908 8935 log.cc:824] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Log is configured to *not* fsync() on all Append() calls
I20260809 16:40:14.175156 8930 raft_consensus.cc:348] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.175323 8930 raft_consensus.cc:374] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:14.175370 8930 raft_consensus.cc:729] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfb02d1b9da4c34b00ab99348267813, State: Initialized, Role: FOLLOWER
I20260809 16:40:14.175506 8930 consensus_queue.cc:260] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.175783 8935 tablet_bootstrap.cc:492] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: No bootstrap required, opened a new log
I20260809 16:40:14.175793 8930 ts_tablet_manager.cc:1428] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 16:40:14.175875 8935 ts_tablet_manager.cc:1397] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:40:14.175930 8594 heartbeater.cc:503] Master 127.28.62.190:35809 was elected leader, sending a full tablet report...
I20260809 16:40:14.177727 8935 raft_consensus.cc:348] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.177884 8935 raft_consensus.cc:374] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:40:14.177937 8935 raft_consensus.cc:729] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedfba342ff24249aefa978cc2a61dfb, State: Initialized, Role: FOLLOWER
I20260809 16:40:14.178072 8935 consensus_queue.cc:260] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.178431 8935 ts_tablet_manager.cc:1428] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:40:14.178459 8758 heartbeater.cc:503] Master 127.28.62.190:35809 was elected leader, sending a full tablet report...
W20260809 16:40:14.199805 8759 tablet.cc:2369] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:40:14.301985 8595 tablet.cc:2369] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:40:14.375168 8939 raft_consensus.cc:482] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:40:14.375329 8939 raft_consensus.cc:504] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.375801 8939 leader_election.cc:288] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe7e088381c4c018224a8165ad2c493 (127.28.62.131:36579), ebfb02d1b9da4c34b00ab99348267813 (127.28.62.129:42209)
W20260809 16:40:14.379321 8910 tablet.cc:2369] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:40:14.380080 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.376382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:50467 (local address 127.28.62.129:42209)
0809 16:40:14.376440 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.376442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.376469 (+ 27us) server_negotiation.cc:407] Connection header received
0809 16:40:14.376565 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.376566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.376577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.376631 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:40:14.376789 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.377521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379244 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.379708 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379797 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.379806 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.379971 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.380012 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 16:40:14.380110 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.376317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:36273 (local address 127.28.62.131:36579)
0809 16:40:14.376370 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.376372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.376384 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:40:14.376416 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.376417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.376428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.376513 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:40:14.376771 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.377521 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379218 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.379733 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379832 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.379841 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.380007 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.380043 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:40:14.380105 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.376247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42209 (local address 127.28.62.130:50467)
0809 16:40:14.376423 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.376431 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.376475 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.376653 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.376654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.376655 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:40:14.376717 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.376718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.377647 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.377648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.379113 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.379116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.379796 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.379879 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.379883 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.379961 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.380006 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":57,"thread_start_us":123,"threads_started":1}
I20260809 16:40:14.380117 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.376099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36579 (local address 127.28.62.130:36273)
0809 16:40:14.376160 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.376173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:40:14.376224 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:40:14.376533 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:40:14.376534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:40:14.376536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:40:14.376655 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.376657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.377640 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.377641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.379113 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:40:14.379116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.379831 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.379832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:40:14.379925 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.379928 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:40:14.380000 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 16:40:14.380041 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":100}
I20260809 16:40:14.380867 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f618145bb0e4450bccb625ba3f9e9af" candidate_uuid: "fedfba342ff24249aefa978cc2a61dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7e088381c4c018224a8165ad2c493" is_pre_election: true
I20260809 16:40:14.380869 8544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f618145bb0e4450bccb625ba3f9e9af" candidate_uuid: "fedfba342ff24249aefa978cc2a61dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfb02d1b9da4c34b00ab99348267813" is_pre_election: true
I20260809 16:40:14.381033 8544 raft_consensus.cc:2454] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fedfba342ff24249aefa978cc2a61dfb in term 0.
I20260809 16:40:14.381033 8854 raft_consensus.cc:2454] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fedfba342ff24249aefa978cc2a61dfb in term 0.
I20260809 16:40:14.381248 8624 leader_election.cc:302] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebfb02d1b9da4c34b00ab99348267813, fedfba342ff24249aefa978cc2a61dfb; no voters:
I20260809 16:40:14.381369 8939 raft_consensus.cc:2781] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:40:14.381425 8939 raft_consensus.cc:482] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:40:14.381449 8939 raft_consensus.cc:3037] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:14.382146 8939 raft_consensus.cc:504] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.382292 8939 leader_election.cc:288] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [CANDIDATE]: Term 1 election: Requested vote from peers cbe7e088381c4c018224a8165ad2c493 (127.28.62.131:36579), ebfb02d1b9da4c34b00ab99348267813 (127.28.62.129:42209)
I20260809 16:40:14.382467 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f618145bb0e4450bccb625ba3f9e9af" candidate_uuid: "fedfba342ff24249aefa978cc2a61dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe7e088381c4c018224a8165ad2c493"
I20260809 16:40:14.382541 8854 raft_consensus.cc:3037] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:14.382513 8544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f618145bb0e4450bccb625ba3f9e9af" candidate_uuid: "fedfba342ff24249aefa978cc2a61dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfb02d1b9da4c34b00ab99348267813"
I20260809 16:40:14.382618 8544 raft_consensus.cc:3037] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:40:14.383422 8854 raft_consensus.cc:2454] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fedfba342ff24249aefa978cc2a61dfb in term 1.
I20260809 16:40:14.383514 8544 raft_consensus.cc:2454] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fedfba342ff24249aefa978cc2a61dfb in term 1.
I20260809 16:40:14.383602 8624 leader_election.cc:302] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbe7e088381c4c018224a8165ad2c493, fedfba342ff24249aefa978cc2a61dfb; no voters:
I20260809 16:40:14.383708 8939 raft_consensus.cc:2781] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:40:14.383858 8939 raft_consensus.cc:686] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 1 LEADER]: Becoming Leader. State: Replica: fedfba342ff24249aefa978cc2a61dfb, State: Running, Role: LEADER
I20260809 16:40:14.383985 8939 consensus_queue.cc:237] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } }
I20260809 16:40:14.384873 8372 catalog_manager.cc:5495] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to fedfba342ff24249aefa978cc2a61dfb (127.28.62.130). New cstate: current_term: 1 leader_uuid: "fedfba342ff24249aefa978cc2a61dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedfba342ff24249aefa978cc2a61dfb" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20260809 16:40:14.390547 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.388142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.28.62.130:37013)
0809 16:40:14.388189 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:40:14.388192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:40:14.388228 (+ 36us) server_negotiation.cc:407] Connection header received
0809 16:40:14.388260 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:40:14.388261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:40:14.388285 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:40:14.388362 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:40:14.388543 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.388998 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.389748 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:40:14.389943 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:40:14.390017 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:40:14.390191 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:40:14.390297 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:40:14.390384 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 16:40:14.390425 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 16:40:14.390445 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:40:14.392460 8544 raft_consensus.cc:1261] T 7f618145bb0e4450bccb625ba3f9e9af P ebfb02d1b9da4c34b00ab99348267813 [term 1 FOLLOWER]: Refusing update from remote peer fedfba342ff24249aefa978cc2a61dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:40:14.392459 8854 raft_consensus.cc:1261] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 1 FOLLOWER]: Refusing update from remote peer fedfba342ff24249aefa978cc2a61dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:40:14.392861 8941 consensus_queue.cc:1035] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebfb02d1b9da4c34b00ab99348267813" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:40:14.392994 8939 consensus_queue.cc:1035] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe7e088381c4c018224a8165ad2c493" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:40:14.397589 8374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:40:14.397689 8374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:40:14.398476 8374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0de2d52598a4879810d073506f75c6f: Sending DeleteTablet for 3 replicas of tablet 7f618145bb0e4450bccb625ba3f9e9af
I20260809 16:40:14.398840 8679 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:40:14 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:55570
I20260809 16:40:14.398890 8519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:40:14 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:56826
I20260809 16:40:14.399271 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8445
I20260809 16:40:14.399268 8830 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f618145bb0e4450bccb625ba3f9e9af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:40:14 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:57652
I20260809 16:40:14.404812 8956 tablet_replica.cc:330] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: stopping tablet replica
I20260809 16:40:14.404897 8958 tablet_replica.cc:330] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: stopping tablet replica
I20260809 16:40:14.405031 8958 raft_consensus.cc:2229] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:40:14.405035 8956 raft_consensus.cc:2229] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:40:14.405156 8958 raft_consensus.cc:2258] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:40:14.405282 8956 raft_consensus.cc:2258] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:40:14.406138 8958 ts_tablet_manager.cc:1905] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:40:14.407433 8956 ts_tablet_manager.cc:1905] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:40:14.408046 8958 ts_tablet_manager.cc:1918] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:40:14.408129 8958 log.cc:1196] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/7f618145bb0e4450bccb625ba3f9e9af
I20260809 16:40:14.408409 8958 ts_tablet_manager.cc:1939] T 7f618145bb0e4450bccb625ba3f9e9af P cbe7e088381c4c018224a8165ad2c493: Deleting consensus metadata
I20260809 16:40:14.408870 8354 catalog_manager.cc:4841] TS cbe7e088381c4c018224a8165ad2c493 (127.28.62.131:36579): tablet 7f618145bb0e4450bccb625ba3f9e9af (table test-table [id=6aaa51e19a9b40a0ba4d8589007d65ae]) successfully deleted
I20260809 16:40:14.409394 8956 ts_tablet_manager.cc:1918] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:40:14.409471 8956 log.cc:1196] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/7f618145bb0e4450bccb625ba3f9e9af
I20260809 16:40:14.409746 8956 ts_tablet_manager.cc:1939] T 7f618145bb0e4450bccb625ba3f9e9af P fedfba342ff24249aefa978cc2a61dfb: Deleting consensus metadata
I20260809 16:40:14.410140 8354 catalog_manager.cc:4841] TS fedfba342ff24249aefa978cc2a61dfb (127.28.62.130:37013): tablet 7f618145bb0e4450bccb625ba3f9e9af (table test-table [id=6aaa51e19a9b40a0ba4d8589007d65ae]) successfully deleted
W20260809 16:40:14.410964 8354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:42209 (error 108)
I20260809 16:40:14.411392 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8602
I20260809 16:40:14.411549 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:40:14.411405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42209 (local address 127.0.0.1:56842)
0809 16:40:14.411472 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 16:40:14.411515 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 16:40:14.417110 8354 catalog_manager.cc:4584] TS ebfb02d1b9da4c34b00ab99348267813 (127.28.62.129:42209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f618145bb0e4450bccb625ba3f9e9af: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42209: connect: Connection refused (error 111)
I20260809 16:40:14.421211 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8763
I20260809 16:40:14.430675 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 8337
2026-08-09T16:40:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2259 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:40:14 dist-test-slave-wl2h krb5kdc[8962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:40:14 dist-test-slave-wl2h krb5kdc[8962](info): set up 2 sockets
Aug 09 16:40:14 dist-test-slave-wl2h krb5kdc[8962](info): commencing operation
krb5kdc: starting...
W20260809 16:40:15.572976 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:40:15 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:40:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangeradmin_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangeradmin_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangerlookup_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangerlookup_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
I20260809 16:40:15.753670 28922 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskb9mnon/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/postgres -l logfile start
2026-08-09 16:40:18.547 UTC [8999] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 16:40:18.547 UTC [8999] LOG: listening on IPv4 address "127.28.62.148", port 44285
2026-08-09 16:40:18.551 UTC [8999] LOG: listening on Unix socket "/tmp/.s.PGSQL.44285"
2026-08-09 16:40:18.575 UTC [9002] LOG: database system was shut down at 2026-08-09 16:40:16 UTC
2026-08-09 16:40:18.581 UTC [8999] LOG: database system is ready to accept connections
I20260809 16:40:19.642287 28922 mini_postgres.cc:96] Postgres bound to 44285
2026-08-09 16:40:19.650 UTC [9012] FATAL: database "slave" does not exist
127.28.62.148:44285 - accepting connections
I20260809 16:40:19.651791 28922 mini_ranger.cc:161] Starting Ranger...
I20260809 16:40:19.670226 28922 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 16:40:20.247803 28922 mini_ranger.cc:87] Created ranger Postgres database
I20260809 16:40:20.247901 28922 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin
2026-08-09 16:40:20,607 [I] DB FLAVOR :POSTGRES
2026-08-09 16:40:20,608 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 16:40:20,608 [I] Checking connection..
2026-08-09 16:40:20,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 16:40:20,885 [I] Checking connection passed.
2026-08-09 16:40:20,886 [I] --------- Verifying version history table ---------
2026-08-09 16:40:20,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:40:21,146 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 16:40:21,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:40:21,415 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 16:40:21,415 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 16:40:21,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 16:40:21.695 UTC [9087] WARNING: there is no transaction in progress
2026-08-09 16:40:21,704 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 16:40:21,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:40:21,968 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 16:40:21,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 16:40:21,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 16:40:22,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 16:40:22,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wl2h.c.gcp-upstream.internal','N') ;"
2026-08-09 16:40:22,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 16:40:22,980 [I] Table x_portal_user does not exist in database ranger
2026-08-09 16:40:22,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 16:40:23,260 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 16:40:23,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 16:40:23,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 16:40:23,794 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 16:40:23,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 16:40:24.845 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:24.864 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:24.882 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:24.900 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:24.918 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:24.941 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.011 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.019 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.035 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.046 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.057 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.074 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.085 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.092 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.101 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25.109 UTC [9214] WARNING: there is no transaction in progress
2026-08-09 16:40:25,834 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 16:40:25,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 16:40:26,077 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 16:40:26,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 16:40:26,355 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 16:40:26,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 16:40:26,625 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 16:40:26,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 16:40:26,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 16:40:27,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wl2h.c.gcp-upstream.internal';"
2026-08-09 16:40:27,422 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 16:40:27,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 16:40:27,671 [I] DB_PATCHES have already been applied
I20260809 16:40:27.677767 28922 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskb9mnon/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 16:40:27.677847 28922 mini_ranger.cc:193] Using host: 127.28.62.148
I20260809 16:40:27.681677 28922 mini_ranger.cc:227] Ranger admin URL: http://127.28.62.148:51025
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:40:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 16:40:28.891943 28922 mini_ranger.cc:160] Time spent starting Ranger: real 9.240s user 0.001s sys 0.011s
Aug 09, 2026 4:40:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:40:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 4:40:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:40:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:40:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 16:40:50.757380 28922 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangerkms_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/rangerkms_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.62.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.62.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 16:40:50 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293650, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 16:40:50.865484 28922 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 16:40:50.876395 28922 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 16:40:51.865634 28922 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 16:40:51.865728 28922 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms
I20260809 16:40:51.865749 28922 mini_ranger_kms.cc:227] Using postgres at 127.28.62.148:44285
2026-08-09 16:40:52,498 [I] DB FLAVOR :POSTGRES
2026-08-09 16:40:52,499 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 16:40:52,499 [I] Checking connection
2026-08-09 16:40:52,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 16:40:52,802 [I] connection success
2026-08-09 16:40:52,802 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 16:40:52,802 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 16:40:52,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 16:40:53,066 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 16:40:53,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 16:40:53,067 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 16:40:53,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:44285/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 16:40:53,348 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 16:40:53.871723 28922 mini_ranger_kms.cc:310] Created kms service
I20260809 16:40:54.105839 28922 mini_ranger_kms.cc:326] Created kudu user
I20260809 16:40:54.153337 28922 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 16:40:54 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293650, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:40:54.577921 28922 mini_ranger_kms.cc:384] Added ranger policy
I20260809 16:40:54.578032 28922 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms:/tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 16:40:54.578052 28922 mini_ranger_kms.cc:240] Using host: 127.28.62.148
I20260809 16:40:54.582693 28922 mini_ranger_kms.cc:276] Ranger KMS PID: 9444
I20260809 16:40:54.582834 28922 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.62.148:45579
Aug 09, 2026 4:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:40:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:40:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 16:40:57.108588 28922 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.243s user 0.020s sys 0.317s
I20260809 16:40:57.108711 28922 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 16:40:57.108785 28922 mini_ranger_kms.cc:401] 127.28.62.148:45579/kms/v1/keys
Aug 09 16:40:57 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786293657, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 16:40:57 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293657, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
Aug 09 16:40:58 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293657, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
Aug 09 16:41:00 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:41:00.643261 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34239
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:34239
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:45579/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:00.757015 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:00.757283 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:00.757323 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:00.758490 9503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:00.758556 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:00.758581 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:00.758612 9503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:00.758641 9503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:00.760390 9503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.62.148:45579/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:34239
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:00.760669 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:00.760898 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:00.766345 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:00.766556 9503 server_base.cc:1034] running on GCE node
W20260809 16:41:00.766338 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:00.766335 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:00.767055 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:00.767390 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:00.768559 9503 hybrid_clock.cc:648] HybridClock initialized: now 1786293660768535 us; error 44 us; skew 500 ppm
Aug 09 16:41:00 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:00.771811 9503 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:00.772294 9503 webserver.cc:460] Webserver started at http://127.28.62.190:38363/ using document root <none> and password file <none>
I20260809 16:41:00.772495 9503 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:00.772531 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:00.772624 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:00 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:00.820927 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "b3ad8432b17845a8b702ceeb53c6a0db"
format_stamp: "Formatted at 2026-08-09 16:41:00 on dist-test-slave-wl2h"
server_key: "7af20b920f6289836d41dbabcddd8839"
server_key_iv: "2ad5bf4a2e733c33e97139393f7a009e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:00.821393 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "b3ad8432b17845a8b702ceeb53c6a0db"
format_stamp: "Formatted at 2026-08-09 16:41:00 on dist-test-slave-wl2h"
server_key: "7af20b920f6289836d41dbabcddd8839"
server_key_iv: "2ad5bf4a2e733c33e97139393f7a009e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:00.823231 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260809 16:41:00.835980 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:00.836220 9503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:00.836329 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "b3ad8432b17845a8b702ceeb53c6a0db"
format_stamp: "Formatted at 2026-08-09 16:41:00 on dist-test-slave-wl2h"
server_key: "7af20b920f6289836d41dbabcddd8839"
server_key_iv: "2ad5bf4a2e733c33e97139393f7a009e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:00.836452 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:00.856524 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:00.858649 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:00.858832 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:00.866189 9503 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:34239
I20260809 16:41:00.866278 9582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:34239 every 8 connection(s)
I20260809 16:41:00.866653 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:00.867516 9583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:00.870319 9583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: Bootstrap starting.
I20260809 16:41:00.871310 9583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:00.871523 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 9503
I20260809 16:41:00.871634 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:41:00.871873 28922 external_mini_cluster.cc:1428] Setting key 50d821b82548a3a9476bf181e7f7a213
I20260809 16:41:00.871951 9583 log.cc:824] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:00.872972 9583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: No bootstrap required, opened a new log
I20260809 16:41:00.875689 9583 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } }
I20260809 16:41:00.875859 9583 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:00.875909 9583 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ad8432b17845a8b702ceeb53c6a0db, State: Initialized, Role: FOLLOWER
I20260809 16:41:00.876106 9583 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } }
I20260809 16:41:00.876232 9583 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:41:00.876276 9583 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:41:00.876327 9583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:00.877022 9583 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } }
I20260809 16:41:00.877148 9583 leader_election.cc:302] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ad8432b17845a8b702ceeb53c6a0db; no voters:
I20260809 16:41:00.877350 9583 leader_election.cc:288] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:41:00.877441 9588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:00.877599 9588 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [term 1 LEADER]: Becoming Leader. State: Replica: b3ad8432b17845a8b702ceeb53c6a0db, State: Running, Role: LEADER
I20260809 16:41:00.877761 9583 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:00.878134 9588 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } }
I20260809 16:41:00.879668 9589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ad8432b17845a8b702ceeb53c6a0db. Latest consensus state: current_term: 1 leader_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } } }
I20260809 16:41:00.879767 9589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:00.880226 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad8432b17845a8b702ceeb53c6a0db" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34239 } } }
I20260809 16:41:00.880326 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:00.880590 9605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 09 16:41:00 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:00.880864 9583 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 16:41:00.881686 9605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:00.890851 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:00.872469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.28.62.190:34239)
0809 16:41:00.872662 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:00.872672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:00.872718 (+ 46us) server_negotiation.cc:407] Connection header received
0809 16:41:00.873321 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:00.873328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:00.873380 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:00.873504 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:00.874573 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:00.876014 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:00.876934 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:00.877689 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:00.877811 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:00.881469 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:00.881473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:00.881473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:00.881485 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:00.888637 (+ 7152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:00.889165 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:00.889168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:00.889169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:00.889306 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:00.889585 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:00.889586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:00.889586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:00.889956 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:00.890150 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:00.890244 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:41:00.890517 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260809 16:41:00.892112 9605 catalog_manager.cc:1347] Generated new cluster ID: 228e7a72bc7c41809a95cd316f1ac9f4
I20260809 16:41:00.892463 9605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:00.900323 9605 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:41:00.901221 9605 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:00.906912 9605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: Generated new TSK 0
I20260809 16:41:00.907188 9605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:41:01.732151 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:45579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:01.875671 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:01.875954 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:01.875993 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:01.877933 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:01.878001 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:01.878046 9640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:41:01.880528 9640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.62.148:45579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:01.880800 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:01.881143 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:01.886729 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:01.886804 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:01.886895 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:01.887264 9640 server_base.cc:1034] running on GCE node
I20260809 16:41:01.887486 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:01.887770 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:01.888952 9640 hybrid_clock.cc:648] HybridClock initialized: now 1786293661888929 us; error 38 us; skew 500 ppm
Aug 09 16:41:01 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:01.892850 9640 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:41:01.893428 9640 webserver.cc:460] Webserver started at http://127.28.62.129:39633/ using document root <none> and password file <none>
I20260809 16:41:01.893764 9640 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:01.893836 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:01.894001 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:01 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:01.910173 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "384c7ae07e134ae4999fa8e7db802357"
format_stamp: "Formatted at 2026-08-09 16:41:01 on dist-test-slave-wl2h"
server_key: "4ae1711d351a05d17699ff3303b4163f"
server_key_iv: "45a36027be4e9dd181eda3e35e834b5d"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:01.910820 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "384c7ae07e134ae4999fa8e7db802357"
format_stamp: "Formatted at 2026-08-09 16:41:01 on dist-test-slave-wl2h"
server_key: "4ae1711d351a05d17699ff3303b4163f"
server_key_iv: "45a36027be4e9dd181eda3e35e834b5d"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:01.912955 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 16:41:01.928828 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:01.929112 9640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:01.929236 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "384c7ae07e134ae4999fa8e7db802357"
format_stamp: "Formatted at 2026-08-09 16:41:01 on dist-test-slave-wl2h"
server_key: "4ae1711d351a05d17699ff3303b4163f"
server_key_iv: "45a36027be4e9dd181eda3e35e834b5d"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:01.929350 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:01.949571 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:01.952163 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:01.952327 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:01.952636 9640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:01.953029 9640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:01.953074 9640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:01.953105 9640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:01.953123 9640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:01.965262 9640 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:41851
I20260809 16:41:01.965684 9789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:41851 every 8 connection(s)
I20260809 16:41:01.965850 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:41:01.973944 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 9640
I20260809 16:41:01.974059 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:41:01.974277 28922 external_mini_cluster.cc:1428] Setting key 60cb5b371f302ffb5cb3d519299e3c15
Aug 09 16:41:01 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:01.987291 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:01.966766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34239 (local address 127.28.62.129:55011)
0809 16:41:01.967289 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:01.967341 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:01.968299 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:01.975107 (+ 6808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:01.975113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:01.975525 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:01.975908 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:01.975916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:01.977093 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:01.977094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:01.979056 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:01.979060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:01.979851 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:01.979852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:01.980072 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:01.980982 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:01.980993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:01.983035 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:01.985234 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:01.985242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:01.985245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:01.985588 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:01.985828 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:01.985829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:01.985830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:01.985969 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:01.986332 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:01.986339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:01.986508 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 16:41:01.986599 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 16:41:01.986888 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":194,"threads_started":1}
I20260809 16:41:01.988185 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:01.966799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:55011 (local address 127.28.62.190:34239)
0809 16:41:01.974818 (+ 8019us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:01.974820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:01.974832 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:41:01.974860 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:01.974862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:01.974875 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:01.976022 (+ 1147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:01.976082 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:01.976928 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:01.979336 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:01.979726 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:01.980729 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:01.983153 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:01.983154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:01.983155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:01.983174 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:01.985109 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:01.985682 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:01.985683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:01.985684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:01.985754 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:01.986049 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:01.986050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:01.986050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:01.986246 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:01.987887 (+ 1641us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:01.987936 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 16:41:01.988029 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7992,"thread_start_us":61,"threads_started":1}
I20260809 16:41:01.988798 9790 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34239
I20260809 16:41:01.988897 9790 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:01.989106 9790 heartbeater.cc:511] Master 127.28.62.190:34239 requested a full tablet report, sending...
I20260809 16:41:01.989681 9541 ts_manager.cc:194] Registered new tserver with Master: 384c7ae07e134ae4999fa8e7db802357 (127.28.62.129:41851)
I20260809 16:41:01.990572 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:55011
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:41:02.022974 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:45579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:02.166059 9797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:02.166355 9797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:02.166400 9797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:02.167845 9797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:02.167920 9797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:02.167972 9797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:41:02.169797 9797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.62.148:45579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:02.170173 9797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:02.170459 9797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:02.177441 9807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:02.177382 9806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:02.177750 9809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:02.178584 9797 server_base.cc:1034] running on GCE node
I20260809 16:41:02.178813 9797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:02.179118 9797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:02.180292 9797 hybrid_clock.cc:648] HybridClock initialized: now 1786293662180276 us; error 26 us; skew 500 ppm
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:02.183910 9797 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:41:02.184490 9797 webserver.cc:460] Webserver started at http://127.28.62.130:38105/ using document root <none> and password file <none>
I20260809 16:41:02.184772 9797 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:02.184852 9797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:02.185009 9797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:02.206974 9797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "037bcbb56fd14bb69d15f952937a23af"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "4301ad08805aa8c62aa310d0c30c7102"
server_key_iv: "44aec6eae7a342e1fbab5ccf69feede7"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.207623 9797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "037bcbb56fd14bb69d15f952937a23af"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "4301ad08805aa8c62aa310d0c30c7102"
server_key_iv: "44aec6eae7a342e1fbab5ccf69feede7"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.215219 9797 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260809 16:41:02.235215 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.235452 9797 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260809 16:41:02.235563 9797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "037bcbb56fd14bb69d15f952937a23af"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "4301ad08805aa8c62aa310d0c30c7102"
server_key_iv: "44aec6eae7a342e1fbab5ccf69feede7"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.235718 9797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:02.263240 9797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:02.266691 9797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:02.266954 9797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:02.267401 9797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:02.267940 9797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:02.268045 9797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.268132 9797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:02.268203 9797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.279886 9797 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:41559
I20260809 16:41:02.280530 9797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:41:02.282405 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 9797
I20260809 16:41:02.282518 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:41:02.282739 28922 external_mini_cluster.cc:1428] Setting key 692b8722aa7082ec00893afae9265b28
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:41:02.292589 9948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:41559 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:02.306471 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.293554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:59529 (local address 127.28.62.190:34239)
0809 16:41:02.293611 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.293613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.294323 (+ 710us) server_negotiation.cc:407] Connection header received
0809 16:41:02.295119 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.295121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.295145 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.295171 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:02.296413 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.297187 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.298805 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.299234 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.299284 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.302267 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:02.302269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:02.302269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:02.302277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:02.304553 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.305195 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.305196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.305197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.305257 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.305550 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.305551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.305552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.305716 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:02.305765 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.306227 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.306335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 16:41:02.307484 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.293327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34239 (local address 127.28.62.130:59529)
0809 16:41:02.294174 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.294210 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.294923 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.295436 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.295439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.295844 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:02.296296 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.296302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.297309 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.297310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.298694 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.298698 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.299382 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.299383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.299569 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.300340 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:02.300352 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:02.302163 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:02.304665 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:02.304671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:02.304674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:02.305099 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:02.305331 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:02.305332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:02.305332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:02.305469 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:02.305840 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:02.305848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:02.306031 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.306924 (+ 893us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.307155 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":582,"threads_started":1}
I20260809 16:41:02.308259 9949 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34239
I20260809 16:41:02.308358 9949 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:02.308578 9949 heartbeater.cc:511] Master 127.28.62.190:34239 requested a full tablet report, sending...
I20260809 16:41:02.309048 9541 ts_manager.cc:194] Registered new tserver with Master: 037bcbb56fd14bb69d15f952937a23af (127.28.62.130:41559)
I20260809 16:41:02.309614 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:59529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:41:02.318883 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:45579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:02.451211 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:02.451601 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:02.451697 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:02.453615 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:02.453740 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:02.453832 9954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:41:02.456540 9954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.62.148:45579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:02.457031 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:02.457387 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:02.476073 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:02.476449 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:02.477856 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:02.478163 9954 server_base.cc:1034] running on GCE node
I20260809 16:41:02.478574 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:02.478943 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:02.480223 9954 hybrid_clock.cc:648] HybridClock initialized: now 1786293662480199 us; error 38 us; skew 500 ppm
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:02.484117 9954 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:41:02.484701 9954 webserver.cc:460] Webserver started at http://127.28.62.131:37415/ using document root <none> and password file <none>
I20260809 16:41:02.484998 9954 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:02.485109 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:02.485284 9954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:02.502808 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "f9bc366a1dc34b1db7ab338d604a4ca8"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "d8fc4f30c75f4d13ce39d4e31aefccca"
server_key_iv: "5beb8daf0574f13f4f2076e71884cc7e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.503433 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "f9bc366a1dc34b1db7ab338d604a4ca8"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "d8fc4f30c75f4d13ce39d4e31aefccca"
server_key_iv: "5beb8daf0574f13f4f2076e71884cc7e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.508582 9954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260809 16:41:02.526878 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.527185 9954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 16:41:02.527281 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "f9bc366a1dc34b1db7ab338d604a4ca8"
format_stamp: "Formatted at 2026-08-09 16:41:02 on dist-test-slave-wl2h"
server_key: "d8fc4f30c75f4d13ce39d4e31aefccca"
server_key_iv: "5beb8daf0574f13f4f2076e71884cc7e"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:02.527380 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:02.552636 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:02.555439 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:02.555653 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:02.556000 9954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:02.556493 9954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:02.556586 9954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.556661 9954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:02.556731 9954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:02.569332 9954 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:42397
I20260809 16:41:02.569907 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:41:02.578630 10109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:42397 every 8 connection(s)
I20260809 16:41:02.579761 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 9954
I20260809 16:41:02.579875 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:41:02.580087 28922 external_mini_cluster.cc:1428] Setting key f2d6651aed756739e413fec930c5e6e0
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:02.597975 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.585264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:35651 (local address 127.28.62.190:34239)
0809 16:41:02.585551 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.585553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.585756 (+ 203us) server_negotiation.cc:407] Connection header received
0809 16:41:02.586544 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.586545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.586559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.586590 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:02.587955 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.588621 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.590201 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.590502 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.590550 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.594365 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:02.594366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:02.594367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:02.594377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:02.596294 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.596883 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.596885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.596885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.596947 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.597223 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.597224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.597224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.597361 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:02.597402 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.597786 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.597872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260809 16:41:02.598803 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.585262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34239 (local address 127.28.62.131:35651)
0809 16:41:02.585550 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.585608 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.586418 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.586794 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.586799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.587458 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:02.587829 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.587836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.588764 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.588764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.590092 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.590096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.590653 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.590653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.590869 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.592240 (+ 1371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:02.592251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:02.594224 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:02.596406 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:02.596416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:02.596419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:02.596785 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:02.597015 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:02.597016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:02.597016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:02.597153 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:02.597447 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:02.597453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:02.597626 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.598369 (+ 743us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.598550 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
I20260809 16:41:02.599476 10110 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34239
I20260809 16:41:02.599572 10110 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:02.599759 10110 heartbeater.cc:511] Master 127.28.62.190:34239 requested a full tablet report, sending...
I20260809 16:41:02.600157 9541 ts_manager.cc:194] Registered new tserver with Master: f9bc366a1dc34b1db7ab338d604a4ca8 (127.28.62.131:42397)
I20260809 16:41:02.600626 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:35651
I20260809 16:41:02.602268 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:41:02.612552 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.604119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.28.62.190:34239)
0809 16:41:02.604158 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.604160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.604497 (+ 337us) server_negotiation.cc:407] Connection header received
0809 16:41:02.604590 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.604591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.604602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.604630 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:02.605461 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.606238 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.607356 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.607651 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.607690 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.609273 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:02.609274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:02.609275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:02.609283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:02.611150 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.611678 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.611680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.611680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.611755 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:02.611970 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:02.611971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:02.611971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:02.612117 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:02.612215 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.612385 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.612465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:41:02.614800 9541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:41:02.617065 9541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:41:02.628458 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.620394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:41559 (local address 127.0.0.1:57442)
0809 16:41:02.620859 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.620872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.620942 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.623440 (+ 2498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.623444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.623446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:02.623539 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.623541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.625087 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.625088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.626478 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.626481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.627873 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.627874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.627959 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.627976 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.628085 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.628147 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":39,"thread_start_us":107,"threads_started":1}
I20260809 16:41:02.629194 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.620759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.28.62.130:41559)
0809 16:41:02.623173 (+ 2414us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.623178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.623190 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:41:02.623306 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.623308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.623326 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.623406 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:02.623672 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.624966 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.626625 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.627518 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.627708 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.627830 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.629023 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.629072 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2373,"thread_start_us":127,"threads_started":1}
I20260809 16:41:02.630277 9871 tablet_service.cc:1467] Processing CreateTablet for tablet 56457d6fd8174b96b0d4dab972510136 (DEFAULT_TABLE table=test-table [id=7fa29952213642539be5b653c426cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:02.630759 9871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56457d6fd8174b96b0d4dab972510136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:02.632023 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.620390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:42397 (local address 127.0.0.1:41740)
0809 16:41:02.620826 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.620871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.620942 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.623896 (+ 2954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.623897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.623899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:02.623945 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.623946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.625241 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.625242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.627490 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.627493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.631474 (+ 3981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.631475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.631541 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.631546 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.631596 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.631636 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":95,"threads_started":1}
I20260809 16:41:02.632391 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.620502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.28.62.131:42397)
0809 16:41:02.623662 (+ 3160us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.623666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.623676 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:41:02.623722 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.623724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.623739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.623818 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:02.624017 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.625134 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.627839 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.631279 (+ 3440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.631358 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.631436 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.632259 (+ 823us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.632311 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3109,"thread_start_us":150,"threads_started":1}
I20260809 16:41:02.632997 10036 tablet_service.cc:1467] Processing CreateTablet for tablet 56457d6fd8174b96b0d4dab972510136 (DEFAULT_TABLE table=test-table [id=7fa29952213642539be5b653c426cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:02.633327 10036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56457d6fd8174b96b0d4dab972510136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:02.637914 10131 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Bootstrap starting.
I20260809 16:41:02.639757 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.621056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.28.62.129:41851)
0809 16:41:02.621205 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.621211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.627748 (+ 6537us) server_negotiation.cc:407] Connection header received
0809 16:41:02.628693 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.628704 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.628721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.628775 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:02.628973 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.630019 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.638352 (+ 8333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.639248 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.639300 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.639356 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.639624 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.639673 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260809 16:41:02.640028 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.620662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:41851 (local address 127.0.0.1:60882)
0809 16:41:02.627662 (+ 7000us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.627674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.628589 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.628866 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.628867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.628868 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:02.628904 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.628906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.630119 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.630120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.638209 (+ 8089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.638214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.639422 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.639423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.639522 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.639527 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.639876 (+ 349us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.639930 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6967,"thread_start_us":83,"threads_started":1}
I20260809 16:41:02.640441 10130 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Bootstrap starting.
I20260809 16:41:02.640775 9709 tablet_service.cc:1467] Processing CreateTablet for tablet 56457d6fd8174b96b0d4dab972510136 (DEFAULT_TABLE table=test-table [id=7fa29952213642539be5b653c426cd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:02.641274 9709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56457d6fd8174b96b0d4dab972510136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:02.641793 10131 tablet_bootstrap.cc:654] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:02.642328 10130 tablet_bootstrap.cc:654] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:02.643013 10130 log.cc:824] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:02.643410 10131 log.cc:824] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:02.643996 10130 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: No bootstrap required, opened a new log
I20260809 16:41:02.644096 10130 ts_tablet_manager.cc:1397] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260809 16:41:02.646483 10136 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Bootstrap starting.
I20260809 16:41:02.647233 10130 raft_consensus.cc:348] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.647403 10130 raft_consensus.cc:374] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:02.647434 10130 raft_consensus.cc:729] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037bcbb56fd14bb69d15f952937a23af, State: Initialized, Role: FOLLOWER
I20260809 16:41:02.647588 10130 consensus_queue.cc:260] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.648258 10130 ts_tablet_manager.cc:1428] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260809 16:41:02.648382 9949 heartbeater.cc:503] Master 127.28.62.190:34239 was elected leader, sending a full tablet report...
I20260809 16:41:02.648934 10136 tablet_bootstrap.cc:654] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:02.649900 10136 log.cc:824] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:02.650377 10131 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: No bootstrap required, opened a new log
I20260809 16:41:02.650454 10131 ts_tablet_manager.cc:1397] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20260809 16:41:02.651702 10136 tablet_bootstrap.cc:492] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: No bootstrap required, opened a new log
I20260809 16:41:02.651825 10136 ts_tablet_manager.cc:1397] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 16:41:02.653228 10131 raft_consensus.cc:348] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.653432 10131 raft_consensus.cc:374] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:02.653465 10131 raft_consensus.cc:729] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bc366a1dc34b1db7ab338d604a4ca8, State: Initialized, Role: FOLLOWER
I20260809 16:41:02.653577 10131 consensus_queue.cc:260] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.653829 10131 ts_tablet_manager.cc:1428] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:41:02.653927 10110 heartbeater.cc:503] Master 127.28.62.190:34239 was elected leader, sending a full tablet report...
I20260809 16:41:02.657428 10136 raft_consensus.cc:348] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.657701 10136 raft_consensus.cc:374] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:02.657768 10136 raft_consensus.cc:729] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c7ae07e134ae4999fa8e7db802357, State: Initialized, Role: FOLLOWER
I20260809 16:41:02.658250 10136 consensus_queue.cc:260] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.658978 10136 ts_tablet_manager.cc:1428] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 16:41:02.659036 9790 heartbeater.cc:503] Master 127.28.62.190:34239 was elected leader, sending a full tablet report...
W20260809 16:41:02.719237 9792 tablet.cc:2369] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:41:02.803473 9950 tablet.cc:2369] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:41:02.829176 10111 tablet.cc:2369] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 16:41:02 dist-test-slave-wl2h krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293662, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:02.919960 10141 raft_consensus.cc:482] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:41:02.920188 10141 raft_consensus.cc:504] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.920680 10141 leader_election.cc:288] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037bcbb56fd14bb69d15f952937a23af (127.28.62.130:41559), f9bc366a1dc34b1db7ab338d604a4ca8 (127.28.62.131:42397)
I20260809 16:41:02.926002 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.921707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:52399 (local address 127.28.62.130:41559)
0809 16:41:02.921767 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.921769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.922023 (+ 254us) server_negotiation.cc:407] Connection header received
0809 16:41:02.922131 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.922132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.922141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.922181 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:02.922607 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.923399 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.924906 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.925541 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.925593 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.925602 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.925879 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.925924 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 16:41:02.926530 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.920998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:41559 (local address 127.28.62.129:52399)
0809 16:41:02.921951 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.921964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.922055 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.922255 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.922256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.922315 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:02.922510 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.922512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.923503 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.923504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.924799 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.924802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.925666 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.925667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.925791 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.925795 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.926350 (+ 555us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.926400 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":176,"threads_started":1}
I20260809 16:41:02.927078 9893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56457d6fd8174b96b0d4dab972510136" candidate_uuid: "384c7ae07e134ae4999fa8e7db802357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037bcbb56fd14bb69d15f952937a23af" is_pre_election: true
I20260809 16:41:02.927259 9893 raft_consensus.cc:2454] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384c7ae07e134ae4999fa8e7db802357 in term 0.
I20260809 16:41:02.927651 9659 leader_election.cc:302] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037bcbb56fd14bb69d15f952937a23af, 384c7ae07e134ae4999fa8e7db802357; no voters:
I20260809 16:41:02.927794 10141 raft_consensus.cc:2781] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:41:02.927834 10141 raft_consensus.cc:482] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:41:02.927856 10141 raft_consensus.cc:3037] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:02.928944 10141 raft_consensus.cc:504] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.929103 10141 leader_election.cc:288] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [CANDIDATE]: Term 1 election: Requested vote from peers 037bcbb56fd14bb69d15f952937a23af (127.28.62.130:41559), f9bc366a1dc34b1db7ab338d604a4ca8 (127.28.62.131:42397)
I20260809 16:41:02.929453 9893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56457d6fd8174b96b0d4dab972510136" candidate_uuid: "384c7ae07e134ae4999fa8e7db802357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037bcbb56fd14bb69d15f952937a23af"
I20260809 16:41:02.929538 9893 raft_consensus.cc:3037] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:02.930457 9893 raft_consensus.cc:2454] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384c7ae07e134ae4999fa8e7db802357 in term 1.
I20260809 16:41:02.930717 9659 leader_election.cc:302] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037bcbb56fd14bb69d15f952937a23af, 384c7ae07e134ae4999fa8e7db802357; no voters:
I20260809 16:41:02.930887 10141 raft_consensus.cc:2781] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:02.931069 10141 raft_consensus.cc:686] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [term 1 LEADER]: Becoming Leader. State: Replica: 384c7ae07e134ae4999fa8e7db802357, State: Running, Role: LEADER
I20260809 16:41:02.931176 10141 consensus_queue.cc:237] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } }
I20260809 16:41:02.932111 9541 catalog_manager.cc:5495] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 384c7ae07e134ae4999fa8e7db802357 (127.28.62.129). New cstate: current_term: 1 leader_uuid: "384c7ae07e134ae4999fa8e7db802357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c7ae07e134ae4999fa8e7db802357" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 41851 } health_report { overall_health: HEALTHY } } }
I20260809 16:41:02.935041 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.921406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:46687 (local address 127.28.62.131:42397)
0809 16:41:02.921460 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.921462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.927230 (+ 5768us) server_negotiation.cc:407] Connection header received
0809 16:41:02.927353 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.927355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.927366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.927400 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:02.927648 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.928449 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.933969 (+ 5520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.934553 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.934643 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.934655 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.934896 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.934943 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:41:02.935137 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.921317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:42397 (local address 127.28.62.129:46687)
0809 16:41:02.927136 (+ 5819us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:02.927150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:02.927265 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:02.927471 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:02.927472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:02.927474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:02.927541 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.927542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.928598 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.928599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.931469 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:02.931472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.934715 (+ 3243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.934716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:02.934801 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.934806 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:41:02.934872 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:41:02.934947 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5775,"thread_start_us":161,"threads_started":1}
I20260809 16:41:02.935649 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56457d6fd8174b96b0d4dab972510136" candidate_uuid: "384c7ae07e134ae4999fa8e7db802357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" is_pre_election: true
I20260809 16:41:02.935623 10047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56457d6fd8174b96b0d4dab972510136" candidate_uuid: "384c7ae07e134ae4999fa8e7db802357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8"
I20260809 16:41:02.935824 10046 raft_consensus.cc:2454] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384c7ae07e134ae4999fa8e7db802357 in term 0.
I20260809 16:41:02.982378 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:02.978946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.28.62.129:41851)
0809 16:41:02.979003 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:02.979005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:02.979030 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:41:02.979076 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:02.979077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:02.979097 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:02.979161 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:41:02.979357 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.980076 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.981142 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:02.981420 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:02.981475 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:02.981830 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:41:02.981971 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:41:02.982043 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:02.982274 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 16:41:02.982297 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:41:02.984645 10047 raft_consensus.cc:3037] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:02.984709 9893 raft_consensus.cc:1261] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 1 FOLLOWER]: Refusing update from remote peer 384c7ae07e134ae4999fa8e7db802357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:02.985011 10141 consensus_queue.cc:1035] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037bcbb56fd14bb69d15f952937a23af" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:02.985978 10047 raft_consensus.cc:1261] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 1 FOLLOWER]: Refusing update from remote peer 384c7ae07e134ae4999fa8e7db802357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:02.986480 10141 consensus_queue.cc:1035] T 56457d6fd8174b96b0d4dab972510136 P 384c7ae07e134ae4999fa8e7db802357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bc366a1dc34b1db7ab338d604a4ca8" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:02.987964 10147 mvcc.cc:205] Tried to move back new op lower bound from 7316658843582193664 to 7316658843366686720. Current Snapshot: MvccSnapshot[applied={T|T < 7316658843582193664}]
I20260809 16:41:02.989319 10150 mvcc.cc:205] Tried to move back new op lower bound from 7316658843582193664 to 7316658843366686720. Current Snapshot: MvccSnapshot[applied={T|T < 7316658843582193664}]
I20260809 16:41:02.991793 10148 mvcc.cc:205] Tried to move back new op lower bound from 7316658843582193664 to 7316658843366686720. Current Snapshot: MvccSnapshot[applied={T|T < 7316658843582193664}]
I20260809 16:41:02.993078 9540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:41:02.993172 9540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:41:02.993919 9540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3ad8432b17845a8b702ceeb53c6a0db: Sending DeleteTablet for 3 replicas of tablet 56457d6fd8174b96b0d4dab972510136
I20260809 16:41:02.994360 9871 tablet_service.cc:1514] Processing DeleteTablet for tablet 56457d6fd8174b96b0d4dab972510136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:02 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:57442
I20260809 16:41:02.994480 10036 tablet_service.cc:1514] Processing DeleteTablet for tablet 56457d6fd8174b96b0d4dab972510136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:02 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:41740
I20260809 16:41:02.994537 9709 tablet_service.cc:1514] Processing DeleteTablet for tablet 56457d6fd8174b96b0d4dab972510136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:02 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:60882
I20260809 16:41:02.994980 10161 tablet_replica.cc:330] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: stopping tablet replica
I20260809 16:41:02.995324 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 9640
W20260809 16:41:03.003216 9522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:41851 (error 108)
I20260809 16:41:03.003253 10161 raft_consensus.cc:2229] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:41:03.003403 10161 raft_consensus.cc:2258] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:41:03.003607 10159 tablet_replica.cc:330] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af: stopping tablet replica
I20260809 16:41:03.003755 10159 raft_consensus.cc:2229] T 56457d6fd8174b96b0d4dab972510136 P 037bcbb56fd14bb69d15f952937a23af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:41:03.003818 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 9797
I20260809 16:41:03.004021 10161 ts_tablet_manager.cc:1905] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:41:03.009943 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:03.009798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:41851 (local address 127.0.0.1:60890)
0809 16:41:03.009852 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:03.009889 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 16:41:03.010085 9522 catalog_manager.cc:4584] TS 384c7ae07e134ae4999fa8e7db802357 (127.28.62.129:41851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56457d6fd8174b96b0d4dab972510136: Network error: Client connection negotiation failed: client connection to 127.28.62.129:41851: connect: Connection refused (error 111)
I20260809 16:41:03.010562 10161 ts_tablet_manager.cc:1918] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:41:03.010710 10161 log.cc:1196] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/56457d6fd8174b96b0d4dab972510136
I20260809 16:41:03.011144 10161 ts_tablet_manager.cc:1939] T 56457d6fd8174b96b0d4dab972510136 P f9bc366a1dc34b1db7ab338d604a4ca8: Deleting consensus metadata
I20260809 16:41:03.011708 9522 catalog_manager.cc:4841] TS f9bc366a1dc34b1db7ab338d604a4ca8 (127.28.62.131:42397): tablet 56457d6fd8174b96b0d4dab972510136 (table test-table [id=7fa29952213642539be5b653c426cd7f]) successfully deleted
I20260809 16:41:03.014431 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:03.014310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:41559 (local address 127.0.0.1:57452)
0809 16:41:03.014366 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:03.014391 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.130:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 16:41:03.014475 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 9954
I20260809 16:41:03.022264 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 9503
I20260809 16:41:03.028836 28922 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 16:41:03.376765 28922 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 16:41:03.376915 28922 mini_ranger.cc:66] Stopping Ranger...
I20260809 16:41:03.731757 28922 mini_ranger.cc:68] Stopped Ranger
2026-08-09 16:41:03.732 UTC [8999] LOG: received smart shutdown request
2026-08-09 16:41:03.736 UTC [8999] LOG: background worker "logical replication launcher" (PID 9009) exited with exit code 1
2026-08-09 16:41:03.736 UTC [9003] LOG: shutting down
2026-08-09 16:41:03.932 UTC [8999] LOG: database system is shut down
2026-08-09T16:41:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49615 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:41:04 dist-test-slave-wl2h krb5kdc[10172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:41:04 dist-test-slave-wl2h krb5kdc[10172](info): set up 2 sockets
Aug 09 16:41:04 dist-test-slave-wl2h krb5kdc[10172](info): commencing operation
krb5kdc: starting...
W20260809 16:41:05.183540 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:41:05 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:41:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangeradmin_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangeradmin_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangerlookup_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangerlookup_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
I20260809 16:41:05.373852 28922 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskb9mnon/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/postgres -l logfile start
2026-08-09 16:41:08.071 UTC [10207] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 16:41:08.072 UTC [10207] LOG: listening on IPv4 address "127.28.62.148", port 50719
2026-08-09 16:41:08.076 UTC [10207] LOG: listening on Unix socket "/tmp/.s.PGSQL.50719"
2026-08-09 16:41:08.112 UTC [10210] LOG: database system was shut down at 2026-08-09 16:41:06 UTC
2026-08-09 16:41:08.116 UTC [10207] LOG: database system is ready to accept connections
I20260809 16:41:09.149895 28922 mini_postgres.cc:96] Postgres bound to 50719
2026-08-09 16:41:09.157 UTC [10219] FATAL: database "slave" does not exist
127.28.62.148:50719 - accepting connections
I20260809 16:41:09.157593 28922 mini_ranger.cc:161] Starting Ranger...
I20260809 16:41:09.173472 28922 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 16:41:09.746513 28922 mini_ranger.cc:87] Created ranger Postgres database
I20260809 16:41:09.746608 28922 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin
2026-08-09 16:41:10,040 [I] DB FLAVOR :POSTGRES
2026-08-09 16:41:10,041 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 16:41:10,041 [I] Checking connection..
2026-08-09 16:41:10,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 16:41:10,321 [I] Checking connection passed.
2026-08-09 16:41:10,322 [I] --------- Verifying version history table ---------
2026-08-09 16:41:10,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:41:10,588 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 16:41:10,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:41:10,867 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 16:41:10,868 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 16:41:10,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 16:41:11.126 UTC [10288] WARNING: there is no transaction in progress
2026-08-09 16:41:11,135 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 16:41:11,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 16:41:11,377 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 16:41:11,378 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 16:41:11,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 16:41:11,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 16:41:11,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wl2h.c.gcp-upstream.internal','N') ;"
2026-08-09 16:41:12,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 16:41:12,439 [I] Table x_portal_user does not exist in database ranger
2026-08-09 16:41:12,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 16:41:12,694 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 16:41:12,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 16:41:12,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 16:41:13,226 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 16:41:13,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 16:41:14.225 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.243 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.261 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.277 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.295 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.314 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.372 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.380 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.395 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.405 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.415 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.430 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.440 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.447 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.455 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:14.462 UTC [10416] WARNING: there is no transaction in progress
2026-08-09 16:41:15,178 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 16:41:15,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 16:41:15,431 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 16:41:15,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 16:41:15,705 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 16:41:15,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 16:41:15,981 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 16:41:15,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 16:41:16,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 16:41:16,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wl2h.c.gcp-upstream.internal';"
2026-08-09 16:41:16,742 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 16:41:16,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 16:41:17,032 [I] DB_PATCHES have already been applied
I20260809 16:41:17.038692 28922 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskb9mnon/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 16:41:17.038775 28922 mini_ranger.cc:193] Using host: 127.28.62.148
I20260809 16:41:17.042657 28922 mini_ranger.cc:227] Ranger admin URL: http://127.28.62.148:50613
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 16:41:18.306016 28922 mini_ranger.cc:160] Time spent starting Ranger: real 9.148s user 0.000s sys 0.011s
Aug 09, 2026 4:41:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:41:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 4:41:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:41:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:41:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 16:41:39.506963 28922 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.62.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.62.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangerkms_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.62.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/rangerkms_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.62.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.62.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.62.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/HTTP_127.28.62.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 16:41:39 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293699, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 16:41:39.600455 28922 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 16:41:39.611810 28922 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 16:41:40.555125 28922 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 16:41:40.555248 28922 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms
I20260809 16:41:40.555274 28922 mini_ranger_kms.cc:227] Using postgres at 127.28.62.148:50719
2026-08-09 16:41:40,727 [I] DB FLAVOR :POSTGRES
2026-08-09 16:41:40,727 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 16:41:40,727 [I] Checking connection
2026-08-09 16:41:40,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 16:41:40,995 [I] connection success
2026-08-09 16:41:40,995 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 16:41:40,995 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 16:41:40,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 16:41:41,236 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 16:41:41,237 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 16:41:41,237 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 16:41:41,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.62.148:50719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 16:41:41,529 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 16:41:42.040714 28922 mini_ranger_kms.cc:310] Created kms service
I20260809 16:41:42.270473 28922 mini_ranger_kms.cc:326] Created kudu user
I20260809 16:41:42.318908 28922 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 16:41:42 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293699, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:42.710578 28922 mini_ranger_kms.cc:384] Added ranger policy
I20260809 16:41:42.710811 28922 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms:/tmp/dist-test-taskb9mnon/build/release/bin/postgresql.jar:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskb9mnon/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 16:41:42.710841 28922 mini_ranger_kms.cc:240] Using host: 127.28.62.148
I20260809 16:41:42.715538 28922 mini_ranger_kms.cc:276] Ranger KMS PID: 10618
I20260809 16:41:42.715626 28922 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.62.148:55737
Aug 09, 2026 4:41:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:41:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:41:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:41:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:41:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:41:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 16:41:45 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786293705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:45.219709 28922 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.619s user 0.008s sys 0.149s
I20260809 16:41:45.219827 28922 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 16:41:45.219873 28922 mini_ranger_kms.cc:401] 127.28.62.148:55737/kms/v1/keys
Aug 09 16:41:45 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
Aug 09 16:41:45 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293705, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.62.148@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
Aug 09 16:41:48 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:41:48.174472 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42241
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:42241
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:48.259115 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:48.259358 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:48.259382 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:48.260659 10675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:48.260713 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:48.260728 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:48.260741 10675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:48.260754 10675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:48.262490 10675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.62.148:55737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:42241
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:48.262706 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:48.262894 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:41:48.268635 10675 server_base.cc:1034] running on GCE node
W20260809 16:41:48.268596 10683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:48.268563 10684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:48.268582 10686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:48.269132 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:48.269484 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:48.270669 10675 hybrid_clock.cc:648] HybridClock initialized: now 1786293708270650 us; error 46 us; skew 500 ppm
Aug 09 16:41:48 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293708, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:48.278067 10675 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:48.278630 10675 webserver.cc:460] Webserver started at http://127.28.62.190:44201/ using document root <none> and password file <none>
I20260809 16:41:48.278913 10675 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:48.278978 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:48.279189 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:48 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293708, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:48.327065 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "65ea8cb6327241fd9996c1184e0a7089"
format_stamp: "Formatted at 2026-08-09 16:41:48 on dist-test-slave-wl2h"
server_key: "9b9fd2267e1ae589ecd6a292961c641f"
server_key_iv: "a12fcc9d835243a9b88711862959c65a"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:48.327554 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "65ea8cb6327241fd9996c1184e0a7089"
format_stamp: "Formatted at 2026-08-09 16:41:48 on dist-test-slave-wl2h"
server_key: "9b9fd2267e1ae589ecd6a292961c641f"
server_key_iv: "a12fcc9d835243a9b88711862959c65a"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:48.329452 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 16:41:48.341636 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:48.342015 10675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:48.342132 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "65ea8cb6327241fd9996c1184e0a7089"
format_stamp: "Formatted at 2026-08-09 16:41:48 on dist-test-slave-wl2h"
server_key: "9b9fd2267e1ae589ecd6a292961c641f"
server_key_iv: "a12fcc9d835243a9b88711862959c65a"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:48.342263 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:48.357685 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:48.359768 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:48.359944 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:48.365134 10675 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:42241
I20260809 16:41:48.365218 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:42241 every 8 connection(s)
I20260809 16:41:48.365593 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:48.366324 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:48.368806 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Bootstrap starting.
I20260809 16:41:48.369458 10752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:48.369762 10752 log.cc:824] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:48.370464 10752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: No bootstrap required, opened a new log
I20260809 16:41:48.371348 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 10675
I20260809 16:41:48.371441 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:41:48.371626 28922 external_mini_cluster.cc:1428] Setting key b1b5f80c5430cfa3c6fc88b8bc364e35
I20260809 16:41:48.373016 10752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:48.373183 10752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:48.373251 10752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ea8cb6327241fd9996c1184e0a7089, State: Initialized, Role: FOLLOWER
I20260809 16:41:48.373449 10752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:48.373546 10752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:41:48.373584 10752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:41:48.373656 10752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:48.374369 10752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:48.374498 10752 leader_election.cc:302] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ea8cb6327241fd9996c1184e0a7089; no voters:
I20260809 16:41:48.374712 10752 leader_election.cc:288] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:41:48.374745 10757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:48.375000 10757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 LEADER]: Becoming Leader. State: Replica: 65ea8cb6327241fd9996c1184e0a7089, State: Running, Role: LEADER
I20260809 16:41:48.375226 10757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:48.375121 10752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:48.376765 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ea8cb6327241fd9996c1184e0a7089. Latest consensus state: current_term: 1 leader_uuid: "65ea8cb6327241fd9996c1184e0a7089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } } }
I20260809 16:41:48.376937 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:48.377046 10759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ea8cb6327241fd9996c1184e0a7089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } } }
I20260809 16:41:48.377120 10759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:48.377210 10772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:41:48.377511 10752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 09 16:41:48 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:48.383180 10772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:48.386343 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:48.372184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.28.62.190:42241)
0809 16:41:48.372380 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:48.372389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:48.372421 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:41:48.373021 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:48.373029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:48.373095 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:48.373364 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:48.374098 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:48.375538 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:48.376712 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:48.377346 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:48.377449 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:48.381466 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:48.381510 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:48.381510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:48.381590 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:48.383963 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:48.384999 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:48.385003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:48.385004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:48.385107 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:48.385370 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:48.385370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:48.385371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:48.385697 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:48.385870 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:48.385963 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 16:41:48.386123 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260809 16:41:48.388167 10772 catalog_manager.cc:1347] Generated new cluster ID: 037f8310cab24149b838128f54472f7b
I20260809 16:41:48.388268 10772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:48.417022 10772 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:41:48.417877 10772 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:48.423769 10772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Generated new TSK 0
I20260809 16:41:48.423978 10772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:41:49.162919 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:49.252913 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:49.253153 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:49.253196 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:49.254410 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:49.254455 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:49.254482 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:41:49.256099 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:49.256389 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:49.256647 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:49.261754 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.261816 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.261747 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:49.261973 10804 server_base.cc:1034] running on GCE node
I20260809 16:41:49.262358 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:49.262648 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:49.263849 10804 hybrid_clock.cc:648] HybridClock initialized: now 1786293709263836 us; error 48 us; skew 500 ppm
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:49.267210 10804 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:41:49.267792 10804 webserver.cc:460] Webserver started at http://127.28.62.129:45379/ using document root <none> and password file <none>
I20260809 16:41:49.268121 10804 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:49.268208 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:49.268357 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:49.289342 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "eacfde79100c7ce4296c67d2e34af4b0"
server_key_iv: "c5a31edd3376fb892d3784de9fc6a0d3"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.290115 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "eacfde79100c7ce4296c67d2e34af4b0"
server_key_iv: "c5a31edd3376fb892d3784de9fc6a0d3"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.292220 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 16:41:49.305424 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.305676 10804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:49.305790 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "eacfde79100c7ce4296c67d2e34af4b0"
server_key_iv: "c5a31edd3376fb892d3784de9fc6a0d3"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.305893 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:49.334689 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:49.337342 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:49.337587 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:49.337913 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:49.338590 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:49.338692 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.338798 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:49.338879 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.345649 10804 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:34313
I20260809 16:41:49.346155 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:41:49.348421 10958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:34313 every 8 connection(s)
I20260809 16:41:49.349866 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 10804
I20260809 16:41:49.349978 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:41:49.350217 28922 external_mini_cluster.cc:1428] Setting key c0e5f4533a2656ce03464df8c960de9a
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:41:49.362821 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.349462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:48103 (local address 127.28.62.190:42241)
0809 16:41:49.349624 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:49.349626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:49.349639 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:41:49.352363 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:49.352365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:49.352381 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:49.352455 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:49.353257 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.354015 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.355533 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.355890 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.355986 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.358876 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:49.358877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:49.358878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:49.358886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:49.360975 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.361488 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.361489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.361490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.361767 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.362101 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.362102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.362102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.362229 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:49.362314 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:49.362593 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 16:41:49.362720 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260809 16:41:49.362965 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.349141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.129:48103)
0809 16:41:49.349367 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:49.349424 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:49.350937 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:49.352589 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:49.352593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:49.352857 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:49.353145 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.353150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.354141 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.354142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.355421 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.355424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.356030 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.356031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.356206 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.356930 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:49.356940 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:49.358761 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:49.361095 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.361103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.361106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.361399 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.361851 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.361853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.361853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.361995 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.362305 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:49.362312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:49.362446 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 16:41:49.362563 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 16:41:49.362724 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260809 16:41:49.363766 10959 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:49.363890 10959 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:49.364084 10959 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:49.364735 10711 ts_manager.cc:194] Registered new tserver with Master: 668b2f3f7f4b47adb3e209f4dd27eb8a (127.28.62.129:34313)
I20260809 16:41:49.365613 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:48103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:41:49.388283 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:49.505864 10967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:49.506165 10967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:49.506208 10967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:49.508113 10967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:49.508186 10967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:49.508232 10967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:41:49.510493 10967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:49.510771 10967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:49.511119 10967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 16:41:49.517649 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.517649 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.517618 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:49.518313 10967 server_base.cc:1034] running on GCE node
I20260809 16:41:49.518533 10967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:49.518853 10967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:49.520045 10967 hybrid_clock.cc:648] HybridClock initialized: now 1786293709520028 us; error 58 us; skew 500 ppm
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:49.523509 10967 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:41:49.524053 10967 webserver.cc:460] Webserver started at http://127.28.62.130:35173/ using document root <none> and password file <none>
I20260809 16:41:49.524339 10967 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:49.524405 10967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:49.524547 10967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:49.547386 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "78cabab85f4f4cb3975474ab51b18a29"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "6e383896add16f08adf8ea7c53fe1f89"
server_key_iv: "43ebc74e9ce19ed92c8fd64f0ac56c54"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.547896 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "78cabab85f4f4cb3975474ab51b18a29"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "6e383896add16f08adf8ea7c53fe1f89"
server_key_iv: "43ebc74e9ce19ed92c8fd64f0ac56c54"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.549674 10967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 16:41:49.571830 10987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.572073 10967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 16:41:49.572182 10967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "78cabab85f4f4cb3975474ab51b18a29"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "6e383896add16f08adf8ea7c53fe1f89"
server_key_iv: "43ebc74e9ce19ed92c8fd64f0ac56c54"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.572292 10967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:49.583511 10967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:49.585978 10967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:49.586130 10967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:49.586404 10967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:49.586790 10967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:49.586848 10967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.586885 10967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:49.586907 10967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.594385 10967 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:34047
I20260809 16:41:49.594868 10967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:41:49.595788 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:34047 every 8 connection(s)
I20260809 16:41:49.596518 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 10967
I20260809 16:41:49.596611 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:41:49.596813 28922 external_mini_cluster.cc:1428] Setting key 441212bc87fb452287d2c05679d435a3
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:41:49.610664 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.596736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:46695 (local address 127.28.62.190:42241)
0809 16:41:49.596790 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:49.596793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:49.599587 (+ 2794us) server_negotiation.cc:407] Connection header received
0809 16:41:49.600311 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:49.600312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:49.600330 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:49.600363 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:49.601331 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.602132 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.603676 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.604010 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.604059 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.606990 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:49.606992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:49.606993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:49.607000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:49.608869 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.609479 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.609481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.609481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.609543 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.609832 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.609833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.609834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.609995 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:49.610041 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:49.610440 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 16:41:49.610559 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:41:49.611529 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.596434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.130:46695)
0809 16:41:49.599447 (+ 3013us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:49.599497 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:49.600178 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:49.600561 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:49.600565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:49.600825 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:49.601200 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.601207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.602251 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.602252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.603560 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.603564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.604171 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.604172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.604489 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.605108 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:49.605119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:49.606829 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:49.608966 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.608973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.608976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.609342 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.609617 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.609618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.609618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.609745 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.610106 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:49.610112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:49.610277 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 16:41:49.611088 (+ 811us) client_negotiation.cc:240] Negotiation successful
0809 16:41:49.611288 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2965,"thread_start_us":89,"threads_started":1}
I20260809 16:41:49.612232 11121 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:49.612329 11121 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:49.612512 11121 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:49.612958 10711 ts_manager.cc:194] Registered new tserver with Master: 78cabab85f4f4cb3975474ab51b18a29 (127.28.62.130:34047)
I20260809 16:41:49.613454 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:46695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:41:49.638659 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:49.757169 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:49.757467 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:49.757511 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:49.759275 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:49.759339 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:49.759380 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:41:49.761550 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:49.761799 11126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:49.762081 11126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:49.768535 11136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.768630 11134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:49.768874 11133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:49.769543 11126 server_base.cc:1034] running on GCE node
I20260809 16:41:49.769836 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:49.770141 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:49.771320 11126 hybrid_clock.cc:648] HybridClock initialized: now 1786293709771292 us; error 46 us; skew 500 ppm
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:49.774616 11126 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:41:49.775200 11126 webserver.cc:460] Webserver started at http://127.28.62.131:44047/ using document root <none> and password file <none>
I20260809 16:41:49.775475 11126 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:49.775538 11126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:49.775681 11126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:49.789737 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "74619fe93eb94d99b4abbda241f08a5b"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "5d2c42f656704fb7dcb64deab01f619f"
server_key_iv: "5dbf7251ad8884c1baea3cff8244db09"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.790253 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "74619fe93eb94d99b4abbda241f08a5b"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "5d2c42f656704fb7dcb64deab01f619f"
server_key_iv: "5dbf7251ad8884c1baea3cff8244db09"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.793763 11126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 16:41:49.810989 11146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.811288 11126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 16:41:49.811697 11126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "74619fe93eb94d99b4abbda241f08a5b"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "5d2c42f656704fb7dcb64deab01f619f"
server_key_iv: "5dbf7251ad8884c1baea3cff8244db09"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:49.811908 11126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:49.823958 11126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:49.826294 11126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:49.826463 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:49.826743 11126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:49.827155 11126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:49.827229 11126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.827263 11126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:49.827282 11126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:49.835001 11126 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44267
I20260809 16:41:49.835153 11275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44267 every 8 connection(s)
I20260809 16:41:49.835495 11126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:41:49.840407 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11126
I20260809 16:41:49.840544 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:41:49.840775 28922 external_mini_cluster.cc:1428] Setting key 770668dc7c5a659df69c67c09a354bb5
Aug 09 16:41:49 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293709, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:49.846004 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.836499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:36819 (local address 127.28.62.190:42241)
0809 16:41:49.836569 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:49.836572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:49.836789 (+ 217us) server_negotiation.cc:407] Connection header received
0809 16:41:49.837387 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:49.837389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:49.837401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:49.837450 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:49.838063 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.838570 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.839686 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.839899 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.839983 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.842662 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:49.842663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:49.842663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:49.842672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:49.844390 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.844910 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.844911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.844912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.844976 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:49.845278 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:49.845279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:49.845279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:49.845396 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:49.845474 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:49.845787 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 16:41:49.845873 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 16:41:49.846186 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:49.836462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.131:36819)
0809 16:41:49.836646 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:49.836684 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:49.837279 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:49.837538 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:49.837542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:49.837757 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:49.837952 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.837956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.838682 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.838683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.839585 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:49.839587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:49.840005 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:49.840005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:49.840125 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:49.840705 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:49.840720 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:49.842564 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:49.844503 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.844511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.844514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.844787 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.845048 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:49.845049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:49.845050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:49.845175 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:49.845480 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:49.845487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:49.845635 (+ 148us) client_negotiation.cc:769] Sending connection context
0809 16:41:49.845762 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 16:41:49.845925 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260809 16:41:49.846750 11276 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:49.846868 11276 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:49.847101 11276 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:49.847595 10711 ts_manager.cc:194] Registered new tserver with Master: 74619fe93eb94d99b4abbda241f08a5b (127.28.62.131:44267)
I20260809 16:41:49.847997 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:36819
I20260809 16:41:49.852826 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:41:49.853541 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 10804
I20260809 16:41:49.877249 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 10967
I20260809 16:41:49.886487 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11126
I20260809 16:41:49.905545 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 10675
I20260809 16:41:49.917243 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42241
--webserver_interface=127.28.62.190
--webserver_port=44201
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:42241
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:50.001014 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:50.001273 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:50.001298 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:50.002476 11279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:50.002586 11279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:50.002612 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:50.002638 11279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:50.002652 11279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:50.004438 11279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:42241
--ranger_config_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:50.004654 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:50.004839 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:50.010241 11286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.010243 11287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.010262 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:50.010552 11279 server_base.cc:1034] running on GCE node
I20260809 16:41:50.010897 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:50.011242 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:50.012414 11279 hybrid_clock.cc:648] HybridClock initialized: now 1786293710012391 us; error 36 us; skew 500 ppm
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:50.015347 11279 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:50.015884 11279 webserver.cc:460] Webserver started at http://127.28.62.190:44201/ using document root <none> and password file <none>
I20260809 16:41:50.016157 11279 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:50.016242 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:50.062783 11279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260809 16:41:50.063776 11298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.064000 11279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:50.064111 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "65ea8cb6327241fd9996c1184e0a7089"
format_stamp: "Formatted at 2026-08-09 16:41:48 on dist-test-slave-wl2h"
server_key: "9b9fd2267e1ae589ecd6a292961c641f"
server_key_iv: "a12fcc9d835243a9b88711862959c65a"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:50.064585 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:50.075932 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:50.078163 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:50.078329 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:50.083429 11279 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:42241
I20260809 16:41:50.083554 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:42241 every 8 connection(s)
I20260809 16:41:50.083930 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:50.085270 11358 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:41:50.085943 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Bootstrap starting.
I20260809 16:41:50.094153 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11279
I20260809 16:41:50.094249 11358 log.cc:824] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:50.094528 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:34313
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:42241
--webserver_port=45379
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:41:50.096617 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:41:50.096915 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Bootstrap complete.
I20260809 16:41:50.099629 11358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:50.099920 11358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ea8cb6327241fd9996c1184e0a7089, State: Initialized, Role: FOLLOWER
I20260809 16:41:50.100111 11358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:50.100196 11358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:41:50.100240 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:41:50.100296 11358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:41:50.101349 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:50.101472 11358 leader_election.cc:302] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ea8cb6327241fd9996c1184e0a7089; no voters:
I20260809 16:41:50.101670 11358 leader_election.cc:288] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 16:41:50.101743 11363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 16:41:50.101925 11363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [term 2 LEADER]: Becoming Leader. State: Replica: 65ea8cb6327241fd9996c1184e0a7089, State: Running, Role: LEADER
I20260809 16:41:50.101948 11358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:50.102041 11363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } }
I20260809 16:41:50.102355 11365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65ea8cb6327241fd9996c1184e0a7089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } } }
I20260809 16:41:50.102355 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ea8cb6327241fd9996c1184e0a7089. Latest consensus state: current_term: 2 leader_uuid: "65ea8cb6327241fd9996c1184e0a7089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ea8cb6327241fd9996c1184e0a7089" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42241 } } }
I20260809 16:41:50.102533 11365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:50.102584 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:50.102823 11374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:41:50.103434 11374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:50.103598 11374 catalog_manager.cc:1259] Loaded cluster ID: 037f8310cab24149b838128f54472f7b
I20260809 16:41:50.103645 11374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:50.103957 11358 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 16:41:50.105060 11374 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:50.105265 11374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Loaded TSK: 0
I20260809 16:41:50.105659 11374 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 16:41:50.208732 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:50.209020 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:50.209062 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:50.210947 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:50.211031 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:50.211076 11362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:41:50.213318 11362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:34313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=45379
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:50.213572 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:50.213835 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:50.220201 11404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.220427 11407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.220222 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:50.223097 11362 server_base.cc:1034] running on GCE node
I20260809 16:41:50.223351 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:50.223637 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:50.224820 11362 hybrid_clock.cc:648] HybridClock initialized: now 1786293710224800 us; error 40 us; skew 500 ppm
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:50.228164 11362 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:41:50.228734 11362 webserver.cc:460] Webserver started at http://127.28.62.129:45379/ using document root <none> and password file <none>
I20260809 16:41:50.229054 11362 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:50.229143 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:50.246630 11362 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260809 16:41:50.247431 11419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.247596 11362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:41:50.247666 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "eacfde79100c7ce4296c67d2e34af4b0"
server_key_iv: "c5a31edd3376fb892d3784de9fc6a0d3"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:50.248896 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:50.266530 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:50.268774 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:50.269009 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:50.269345 11362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:50.269735 11362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:50.269804 11362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.269843 11362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:50.269865 11362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.277254 11362 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:34313
I20260809 16:41:50.277349 11549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:34313 every 8 connection(s)
I20260809 16:41:50.277714 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:41:50.282402 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11362
I20260809 16:41:50.282749 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:34047
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:42241
--webserver_port=35173
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:50.294987 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.278566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.129:46295)
0809 16:41:50.278761 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:50.278814 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:50.279506 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:50.283685 (+ 4179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:50.283690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:50.283910 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:50.284181 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.284187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.285553 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.285554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.286338 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.286340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.286979 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.286981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.287145 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.287736 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:50.287767 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:50.289463 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:50.292978 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.292988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.292991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.293339 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.293706 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.293707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.293707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.293838 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.294268 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:50.294274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:50.294413 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 16:41:50.294518 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 16:41:50.294691 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260809 16:41:50.294974 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.278927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:46295 (local address 127.28.62.190:42241)
0809 16:41:50.283230 (+ 4303us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:50.283239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:50.283302 (+ 63us) server_negotiation.cc:407] Connection header received
0809 16:41:50.283429 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:50.283434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:50.283481 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:50.283643 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:50.284329 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.285433 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.286431 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.286848 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.286984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.289619 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:50.289623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:50.289624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:50.289634 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:50.292691 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.293470 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.293475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.293475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.293628 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.293919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.293921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.293921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.294185 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:50.294285 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:50.294606 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 16:41:50.294746 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4241,"thread_start_us":138,"threads_started":1}
I20260809 16:41:50.296128 11550 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:50.296253 11550 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.296447 11550 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.297343 11550 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:42241 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 16:41:50.297415 11550 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.297475 11550 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:50.297772 11550 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.297848 11550 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.298205 11550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:41:50.402237 11556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:50.402566 11556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:50.402612 11556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:50.404175 11556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:50.404253 11556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:50.404304 11556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:41:50.406461 11556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:34047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=35173
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:50.406751 11556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:50.407055 11556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:50.413250 11566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.413218 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.414283 11564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:50.414611 11556 server_base.cc:1034] running on GCE node
I20260809 16:41:50.414841 11556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:50.415177 11556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:50.416394 11556 hybrid_clock.cc:648] HybridClock initialized: now 1786293710416351 us; error 28 us; skew 500 ppm
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:50.419488 11556 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:41:50.420033 11556 webserver.cc:460] Webserver started at http://127.28.62.130:35173/ using document root <none> and password file <none>
I20260809 16:41:50.420297 11556 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:50.420385 11556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:50.441445 11556 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260809 16:41:50.442656 11577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.442878 11556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:41:50.442965 11556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "78cabab85f4f4cb3975474ab51b18a29"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "6e383896add16f08adf8ea7c53fe1f89"
server_key_iv: "43ebc74e9ce19ed92c8fd64f0ac56c54"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:50.443487 11556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:50.452978 11556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:50.455503 11556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:50.455695 11556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:50.455955 11556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:50.456398 11556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:50.456457 11556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.456492 11556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:50.456521 11556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.464972 11556 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:34047
I20260809 16:41:50.465514 11556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:41:50.470893 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11556
I20260809 16:41:50.471254 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.62.148:55737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:44267
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:42241
--webserver_port=44047
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:41:50.479878 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:34047 every 8 connection(s)
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:50.485108 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.466403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:48039 (local address 127.28.62.190:42241)
0809 16:41:50.466456 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:50.466458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:50.473452 (+ 6994us) server_negotiation.cc:407] Connection header received
0809 16:41:50.474077 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:50.474079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:50.474094 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:50.474129 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:50.475177 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.475975 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.477598 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.477926 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.477970 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.481810 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:50.481812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:50.481812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:50.481821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:50.483629 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.484081 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.484082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.484082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.484148 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.484418 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.484419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.484419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.484548 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:50.484627 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:50.484923 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 16:41:50.485014 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 16:41:50.485520 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.466366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.130:48039)
0809 16:41:50.473264 (+ 6898us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:50.473321 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:50.473939 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:50.474335 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:50.474339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:50.474585 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:50.475051 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.475057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.476104 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.476105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.477491 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.477494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.478085 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.478086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.478229 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.478900 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:50.478911 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:50.481671 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:50.483741 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.483748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.483750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.483990 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.484226 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.484227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.484227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.484342 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.484617 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:50.484624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:50.484781 (+ 157us) client_negotiation.cc:769] Sending connection context
0809 16:41:50.484889 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 16:41:50.485048 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6843,"thread_start_us":85,"threads_started":1}
I20260809 16:41:50.486065 11706 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:50.486500 11706 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.486899 11706 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.487866 11706 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:42241 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 16:41:50.487977 11706 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.488073 11706 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:50.488435 11706 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.488523 11706 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.488795 11706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:41:50.604466 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:50.604759 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:50.604805 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:50.606516 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:50.606583 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:50.606626 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:41:50.608999 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.62.148:55737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=44047
--tserver_master_addrs=127.28.62.190:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:50.609274 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:50.609576 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:50.615863 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.615906 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:50.616293 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:50.617137 11709 server_base.cc:1034] running on GCE node
I20260809 16:41:50.617385 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:50.617666 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:50.618851 11709 hybrid_clock.cc:648] HybridClock initialized: now 1786293710618832 us; error 32 us; skew 500 ppm
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:50.621515 11709 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:41:50.622064 11709 webserver.cc:460] Webserver started at http://127.28.62.131:44047/ using document root <none> and password file <none>
I20260809 16:41:50.622360 11709 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:50.622457 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:50.644251 11709 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260809 16:41:50.644995 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.645146 11709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:50.645231 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "74619fe93eb94d99b4abbda241f08a5b"
format_stamp: "Formatted at 2026-08-09 16:41:49 on dist-test-slave-wl2h"
server_key: "5d2c42f656704fb7dcb64deab01f619f"
server_key_iv: "5dbf7251ad8884c1baea3cff8244db09"
server_key_version: "kuduclusterkey@0"
I20260809 16:41:50.646333 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:50.676188 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:50.678718 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:50.678918 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:50.679185 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:50.679577 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:50.679631 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.679669 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:50.679699 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:50.686348 11709 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44267
I20260809 16:41:50.686859 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:41:50.689515 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44267 every 8 connection(s)
I20260809 16:41:50.693337 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11709
Aug 09 16:41:50 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293710, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:50.701644 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.690778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:39299 (local address 127.28.62.190:42241)
0809 16:41:50.690831 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:50.690833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:50.690851 (+ 18us) server_negotiation.cc:407] Connection header received
0809 16:41:50.691902 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:50.691904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:50.691917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:50.691949 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:50.692829 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.693561 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.695277 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.695557 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.695601 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.698188 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:50.698190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:50.698190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:50.698198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:50.699929 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.700445 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.700446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.700446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.700508 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.700795 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.700797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.700797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.701017 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:50.701066 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:50.701455 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 16:41:50.701546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:41:50.702409 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.690515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:42241 (local address 127.28.62.131:39299)
0809 16:41:50.690716 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:50.690776 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:50.691767 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:50.692170 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:50.692174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:50.692425 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:50.692699 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.692705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.693863 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.693864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.695164 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:50.695168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.695717 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.695718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:50.695873 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.696540 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:50.696550 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:50.698074 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:50.700035 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.700042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.700044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.700350 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.700579 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:50.700580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:50.700580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:50.700706 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:50.701119 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:50.701125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:50.701289 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 16:41:50.701994 (+ 705us) client_negotiation.cc:240] Negotiation successful
0809 16:41:50.702168 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260809 16:41:50.703198 11863 heartbeater.cc:347] Connected to a master server at 127.28.62.190:42241
I20260809 16:41:50.703296 11863 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.703490 11863 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.704147 11863 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:42241 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 16:41:50.704217 11863 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.704275 11863 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:50.704644 11863 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:50.704716 11863 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
W20260809 16:41:50.705063 11863 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 16:41:50.714524 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:50.693723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.28.62.190:42241)
0809 16:41:50.696182 (+ 2459us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:50.696184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:50.696192 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:41:50.696213 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:50.696214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:50.696226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:50.696293 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:50.708194 (+ 11901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.708707 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.710094 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:50.710304 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:50.710384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:50.711766 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:50.711767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:50.711767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:50.711775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:50.713298 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.713671 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.713672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.713672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.713731 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:50.713947 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:50.713947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:50.713948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:50.714061 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:50.714250 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:50.714296 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 16:41:50.714390 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2429,"thread_start_us":77,"threads_started":1}
Aug 09 16:41:51 dist-test-slave-wl2h krb5kdc[10172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786293711, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:51.298398 11550 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:51.298570 11550 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:51.298955 11317 ts_manager.cc:194] Registered new tserver with Master: 668b2f3f7f4b47adb3e209f4dd27eb8a (127.28.62.129:34313)
I20260809 16:41:51.300045 11317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:46295
I20260809 16:41:51.489015 11706 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:51.489189 11706 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:51.489697 11317 ts_manager.cc:194] Registered new tserver with Master: 78cabab85f4f4cb3975474ab51b18a29 (127.28.62.130:34047)
I20260809 16:41:51.490224 11317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:48039
I20260809 16:41:51.705237 11863 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:51.705389 11863 heartbeater.cc:511] Master 127.28.62.190:42241 requested a full tablet report, sending...
I20260809 16:41:51.705719 11317 ts_manager.cc:194] Registered new tserver with Master: 74619fe93eb94d99b4abbda241f08a5b (127.28.62.131:44267)
I20260809 16:41:51.706216 11317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:39299
Aug 09 16:41:51 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293711, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
I20260809 16:41:51.713553 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:41:51.723042 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.714634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.28.62.190:42241)
0809 16:41:51.714842 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:51.714844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:51.714852 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:41:51.714873 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:51.714874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:51.714886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:51.714951 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:51.715711 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.716491 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.717538 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.717870 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.719712 (+ 1842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.719768 (+ 56us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:51.719769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:51.719769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:51.719774 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:51.721624 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:51.722041 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:51.722042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:51.722042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:51.722108 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:51.722320 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:51.722321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:51.722321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:51.722455 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:51.722554 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:51.722715 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 16:41:51.722804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260809 16:41:51.728453 11317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:41:51.730719 11317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:41:51.745519 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.740335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.28.62.129:34313)
0809 16:41:51.741238 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:51.741243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:51.741252 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:41:51.741284 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:51.741286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:51.741301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:51.741370 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:51.741581 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.742673 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.744283 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.745032 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.745100 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.745156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:51.745409 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 16:41:51.745448 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":72,"threads_started":1}
I20260809 16:41:51.745519 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.740040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:44267 (local address 127.0.0.1:42224)
0809 16:41:51.741073 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:51.741083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:51.741133 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:51.741392 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:51.741395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:51.741397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:51.741472 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.741475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.742764 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.742765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.744182 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.744185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.745212 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.745213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.745280 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.745293 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:41:51.745359 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:41:51.745427 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":83,"thread_start_us":77,"threads_started":1}
I20260809 16:41:51.745798 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.740498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.28.62.131:44267)
0809 16:41:51.740664 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:51.740669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:51.741136 (+ 467us) server_negotiation.cc:407] Connection header received
0809 16:41:51.741190 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:51.741193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:51.741210 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:51.741286 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:51.741565 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.742660 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.744288 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.745032 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.745097 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.745156 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:51.745655 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 16:41:51.745709 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260809 16:41:51.746418 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.739783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34313 (local address 127.0.0.1:48258)
0809 16:41:51.740883 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:51.740926 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:51.740993 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:51.741444 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:51.741445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:51.741446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:51.741493 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.741495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.742767 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.742767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.744182 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.744185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.745196 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.745197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.745280 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.745293 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:41:51.745359 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:41:51.746348 (+ 989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":992,"thread_start_us":92,"threads_started":1}
I20260809 16:41:51.746743 11786 tablet_service.cc:1467] Processing CreateTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 (DEFAULT_TABLE table=test-table [id=c5be63825a2d4b2b8a9b75faa6ddeb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:51.747196 11786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81d3835ec14a0b88ffc5c478545f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:51.747334 11473 tablet_service.cc:1467] Processing CreateTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 (DEFAULT_TABLE table=test-table [id=c5be63825a2d4b2b8a9b75faa6ddeb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:51.747645 11473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81d3835ec14a0b88ffc5c478545f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:51.755237 11902 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Bootstrap starting.
I20260809 16:41:51.755954 11902 tablet_bootstrap.cc:654] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:51.756345 11902 log.cc:824] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:51.756837 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.740748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:34047 (local address 127.0.0.1:53128)
0809 16:41:51.743084 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:51.743096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:51.743163 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:51.751364 (+ 8201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:51.751366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:51.751369 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:51.751415 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.751416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.752600 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.752601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.753648 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:51.753651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.756567 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.756568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:51.756643 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.756649 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:41:51.756710 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:41:51.756759 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2317,"thread_start_us":77,"threads_started":1}
I20260809 16:41:51.756843 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:51.740898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.28.62.130:34047)
0809 16:41:51.751107 (+ 10209us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:51.751112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:51.751124 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:41:51.751170 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:51.751172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:51.751189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:51.751268 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:51.751494 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.752496 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.753744 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:51.756491 (+ 2747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:51.756567 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:51.756634 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:51.756725 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 16:41:51.756763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10175,"thread_start_us":63,"threads_started":1}
I20260809 16:41:51.757752 11629 tablet_service.cc:1467] Processing CreateTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 (DEFAULT_TABLE table=test-table [id=c5be63825a2d4b2b8a9b75faa6ddeb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:51.758081 11629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb81d3835ec14a0b88ffc5c478545f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:51.760807 11903 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Bootstrap starting.
I20260809 16:41:51.761215 11901 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Bootstrap starting.
I20260809 16:41:51.761513 11903 tablet_bootstrap.cc:654] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:51.761840 11903 log.cc:824] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:51.762631 11903 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: No bootstrap required, opened a new log
I20260809 16:41:51.762734 11903 ts_tablet_manager.cc:1397] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:41:51.765758 11903 raft_consensus.cc:348] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.766083 11903 raft_consensus.cc:374] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:51.766181 11903 raft_consensus.cc:729] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cabab85f4f4cb3975474ab51b18a29, State: Initialized, Role: FOLLOWER
I20260809 16:41:51.766299 11901 tablet_bootstrap.cc:654] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:51.766465 11903 consensus_queue.cc:260] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.766934 11903 ts_tablet_manager.cc:1428] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 16:41:51.767122 11706 heartbeater.cc:503] Master 127.28.62.190:42241 was elected leader, sending a full tablet report...
I20260809 16:41:51.768945 11901 log.cc:824] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:51.771870 11902 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: No bootstrap required, opened a new log
I20260809 16:41:51.771958 11902 ts_tablet_manager.cc:1397] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20260809 16:41:51.774577 11902 raft_consensus.cc:348] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.780486 11901 tablet_bootstrap.cc:492] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: No bootstrap required, opened a new log
I20260809 16:41:51.780872 11901 ts_tablet_manager.cc:1397] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20260809 16:41:51.783427 11901 raft_consensus.cc:348] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.783689 11901 raft_consensus.cc:374] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:51.783766 11901 raft_consensus.cc:729] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74619fe93eb94d99b4abbda241f08a5b, State: Initialized, Role: FOLLOWER
I20260809 16:41:51.783968 11901 consensus_queue.cc:260] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.784271 11902 raft_consensus.cc:374] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:51.784322 11901 ts_tablet_manager.cc:1428] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:41:51.784444 11863 heartbeater.cc:503] Master 127.28.62.190:42241 was elected leader, sending a full tablet report...
I20260809 16:41:51.784735 11902 raft_consensus.cc:729] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668b2f3f7f4b47adb3e209f4dd27eb8a, State: Initialized, Role: FOLLOWER
I20260809 16:41:51.784938 11902 consensus_queue.cc:260] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:51.785249 11902 ts_tablet_manager.cc:1428] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260809 16:41:51.785392 11550 heartbeater.cc:503] Master 127.28.62.190:42241 was elected leader, sending a full tablet report...
W20260809 16:41:51.940982 11865 tablet.cc:2369] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:41:51.980908 11707 tablet.cc:2369] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 16:41:51 dist-test-slave-wl2h krb5kdc[10172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293711, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for HTTP/127.28.62.148@KRBTEST.COM
W20260809 16:41:52.029829 11552 tablet.cc:2369] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:41:52.043577 11906 raft_consensus.cc:482] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:41:52.043731 11906 raft_consensus.cc:504] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:52.044323 11906 leader_election.cc:288] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668b2f3f7f4b47adb3e209f4dd27eb8a (127.28.62.129:34313), 74619fe93eb94d99b4abbda241f08a5b (127.28.62.131:44267)
I20260809 16:41:52.049973 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.044741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:33385 (local address 127.28.62.129:34313)
0809 16:41:52.044973 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:52.044975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:52.045649 (+ 674us) server_negotiation.cc:407] Connection header received
0809 16:41:52.045763 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:52.045764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:52.045776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:52.045834 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:52.046128 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.046790 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.048703 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.049216 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.049336 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:52.049348 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:52.049829 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 16:41:52.049880 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260809 16:41:52.050031 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.044468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34313 (local address 127.28.62.130:33385)
0809 16:41:52.045578 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:52.045590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:52.045680 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:52.045924 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:52.045924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:52.045926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:52.046041 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:52.046043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.047429 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.047430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:52.048611 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:52.048614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.049603 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.049604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:52.049720 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:52.049725 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:41:52.049818 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 16:41:52.049889 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":124,"threads_started":1}
I20260809 16:41:52.050577 11498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81d3835ec14a0b88ffc5c478545f31" candidate_uuid: "78cabab85f4f4cb3975474ab51b18a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" is_pre_election: true
I20260809 16:41:52.050820 11498 raft_consensus.cc:2454] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cabab85f4f4cb3975474ab51b18a29 in term 0.
I20260809 16:41:52.051096 11578 leader_election.cc:302] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668b2f3f7f4b47adb3e209f4dd27eb8a, 78cabab85f4f4cb3975474ab51b18a29; no voters:
I20260809 16:41:52.051249 11906 raft_consensus.cc:2781] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:41:52.051308 11906 raft_consensus.cc:482] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:41:52.051331 11906 raft_consensus.cc:3037] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:52.052453 11906 raft_consensus.cc:504] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:52.052666 11906 leader_election.cc:288] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [CANDIDATE]: Term 1 election: Requested vote from peers 668b2f3f7f4b47adb3e209f4dd27eb8a (127.28.62.129:34313), 74619fe93eb94d99b4abbda241f08a5b (127.28.62.131:44267)
I20260809 16:41:52.052857 11498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81d3835ec14a0b88ffc5c478545f31" candidate_uuid: "78cabab85f4f4cb3975474ab51b18a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a"
I20260809 16:41:52.052944 11498 raft_consensus.cc:3037] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:52.054150 11498 raft_consensus.cc:2454] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cabab85f4f4cb3975474ab51b18a29 in term 1.
I20260809 16:41:52.054356 11578 leader_election.cc:302] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668b2f3f7f4b47adb3e209f4dd27eb8a, 78cabab85f4f4cb3975474ab51b18a29; no voters:
I20260809 16:41:52.054476 11906 raft_consensus.cc:2781] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:52.054688 11906 raft_consensus.cc:686] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 1 LEADER]: Becoming Leader. State: Replica: 78cabab85f4f4cb3975474ab51b18a29, State: Running, Role: LEADER
I20260809 16:41:52.054796 11906 consensus_queue.cc:237] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } }
I20260809 16:41:52.055716 11317 catalog_manager.cc:5495] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78cabab85f4f4cb3975474ab51b18a29 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "78cabab85f4f4cb3975474ab51b18a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78cabab85f4f4cb3975474ab51b18a29" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 } health_report { overall_health: UNKNOWN } } }
I20260809 16:41:52.060351 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.044983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:44267 (local address 127.28.62.131:44267)
0809 16:41:52.045028 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:52.045030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:52.049417 (+ 4387us) server_negotiation.cc:407] Connection header received
0809 16:41:52.051152 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:52.051154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:52.051166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:52.051197 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:52.057178 (+ 5981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.057890 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.059299 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.059911 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.059962 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:52.059971 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:52.060235 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 16:41:52.060278 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:41:52.060850 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.044605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:44267 (local address 127.28.62.130:44267)
0809 16:41:52.049340 (+ 4735us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:52.049354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:52.051048 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:52.056964 (+ 5916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:52.056966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:52.056968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:52.057083 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:52.057085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.057994 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.057995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:52.059189 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:52.059192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.060034 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.060035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:52.060150 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:52.060155 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:41:52.060622 (+ 467us) client_negotiation.cc:240] Negotiation successful
0809 16:41:52.060670 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4698,"thread_start_us":125,"threads_started":1}
I20260809 16:41:52.061662 11806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81d3835ec14a0b88ffc5c478545f31" candidate_uuid: "78cabab85f4f4cb3975474ab51b18a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74619fe93eb94d99b4abbda241f08a5b" is_pre_election: true
I20260809 16:41:52.061830 11806 raft_consensus.cc:2454] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cabab85f4f4cb3975474ab51b18a29 in term 0.
I20260809 16:41:52.062007 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb81d3835ec14a0b88ffc5c478545f31" candidate_uuid: "78cabab85f4f4cb3975474ab51b18a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74619fe93eb94d99b4abbda241f08a5b"
I20260809 16:41:52.062089 11805 raft_consensus.cc:3037] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:52.063344 11805 raft_consensus.cc:2454] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cabab85f4f4cb3975474ab51b18a29 in term 1.
I20260809 16:41:52.125168 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.121508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.28.62.130:34047)
0809 16:41:52.121625 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:52.121627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:52.121637 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:41:52.121684 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:52.121685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:52.121710 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:52.121799 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:41:52.122013 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.122771 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.123945 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:52.124234 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:52.124307 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:52.124539 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:41:52.124701 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:41:52.124797 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:52.125062 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 16:41:52.125085 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 16:41:52.127553 11805 raft_consensus.cc:1261] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 1 FOLLOWER]: Refusing update from remote peer 78cabab85f4f4cb3975474ab51b18a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:52.128018 11906 consensus_queue.cc:1035] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74619fe93eb94d99b4abbda241f08a5b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:52.130569 11498 raft_consensus.cc:1261] T fb81d3835ec14a0b88ffc5c478545f31 P 668b2f3f7f4b47adb3e209f4dd27eb8a [term 1 FOLLOWER]: Refusing update from remote peer 78cabab85f4f4cb3975474ab51b18a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:52.131002 11906 consensus_queue.cc:1035] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668b2f3f7f4b47adb3e209f4dd27eb8a" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:52.136008 11918 mvcc.cc:205] Tried to move back new op lower bound from 7316659044871565312 to 7316659044576940032. Current Snapshot: MvccSnapshot[applied={T|T < 7316659044871565312}]
I20260809 16:41:52.136323 11921 mvcc.cc:205] Tried to move back new op lower bound from 7316659044871565312 to 7316659044576940032. Current Snapshot: MvccSnapshot[applied={T|T < 7316659044871565312}]
I20260809 16:41:52.141187 11317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:41:52.141304 11317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:41:52.142253 11317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ea8cb6327241fd9996c1184e0a7089: Sending DeleteTablet for 3 replicas of tablet fb81d3835ec14a0b88ffc5c478545f31
I20260809 16:41:52.142742 11473 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:52 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:48258
I20260809 16:41:52.143328 11629 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:52 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:53128
I20260809 16:41:52.143745 11786 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81d3835ec14a0b88ffc5c478545f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:41:52 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:42224
I20260809 16:41:52.144299 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11362
I20260809 16:41:52.155002 11919 mvcc.cc:205] Tried to move back new op lower bound from 7316659044871565312 to 7316659044576940032. Current Snapshot: MvccSnapshot[applied={T|T < 7316659044871565312}]
I20260809 16:41:52.159981 11930 tablet_replica.cc:330] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: stopping tablet replica
I20260809 16:41:52.160234 11929 tablet_replica.cc:330] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: stopping tablet replica
I20260809 16:41:52.160424 11929 raft_consensus.cc:2229] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:41:52.160640 11930 raft_consensus.cc:2229] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:41:52.160895 11929 raft_consensus.cc:2258] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:41:52.161139 11930 raft_consensus.cc:2258] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:41:52.167524 11929 ts_tablet_manager.cc:1905] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:41:52.168200 11930 ts_tablet_manager.cc:1905] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:41:52.172001 11930 ts_tablet_manager.cc:1918] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:41:52.172263 11929 ts_tablet_manager.cc:1918] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:41:52.172415 11930 log.cc:1196] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/fb81d3835ec14a0b88ffc5c478545f31
I20260809 16:41:52.172555 11929 log.cc:1196] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/fb81d3835ec14a0b88ffc5c478545f31
I20260809 16:41:52.172983 11930 ts_tablet_manager.cc:1939] T fb81d3835ec14a0b88ffc5c478545f31 P 74619fe93eb94d99b4abbda241f08a5b: Deleting consensus metadata
I20260809 16:41:52.173358 11929 ts_tablet_manager.cc:1939] T fb81d3835ec14a0b88ffc5c478545f31 P 78cabab85f4f4cb3975474ab51b18a29: Deleting consensus metadata
I20260809 16:41:52.173955 11301 catalog_manager.cc:4841] TS 74619fe93eb94d99b4abbda241f08a5b (127.28.62.131:44267): tablet fb81d3835ec14a0b88ffc5c478545f31 (table test-table [id=c5be63825a2d4b2b8a9b75faa6ddeb72]) successfully deleted
I20260809 16:41:52.174561 11300 catalog_manager.cc:4841] TS 78cabab85f4f4cb3975474ab51b18a29 (127.28.62.130:34047): tablet fb81d3835ec14a0b88ffc5c478545f31 (table test-table [id=c5be63825a2d4b2b8a9b75faa6ddeb72]) successfully deleted
W20260809 16:41:52.175045 11299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:34313 (error 108)
I20260809 16:41:52.179407 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11556
I20260809 16:41:52.179569 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:52.179418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34313 (local address 127.0.0.1:48266)
0809 16:41:52.179474 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:52.179525 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 16:41:52.179687 11299 catalog_manager.cc:4584] TS 668b2f3f7f4b47adb3e209f4dd27eb8a (127.28.62.129:34313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb81d3835ec14a0b88ffc5c478545f31: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34313: connect: Connection refused (error 111)
I20260809 16:41:52.188941 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11709
I20260809 16:41:52.226038 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11279
I20260809 16:41:52.234560 28922 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 16:41:52.579347 28922 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 16:41:52.579444 28922 mini_ranger.cc:66] Stopping Ranger...
I20260809 16:41:52.924458 28922 mini_ranger.cc:68] Stopped Ranger
2026-08-09 16:41:52.924 UTC [10207] LOG: received smart shutdown request
2026-08-09 16:41:52.929 UTC [10207] LOG: background worker "logical replication launcher" (PID 10216) exited with exit code 1
2026-08-09 16:41:52.929 UTC [10211] LOG: shutting down
2026-08-09 16:41:53.110 UTC [10207] LOG: database system is shut down
2026-08-09T16:41:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49181 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:41:53 dist-test-slave-wl2h krb5kdc[11944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:41:53 dist-test-slave-wl2h krb5kdc[11944](info): set up 2 sockets
Aug 09 16:41:53 dist-test-slave-wl2h krb5kdc[11944](info): commencing operation
krb5kdc: starting...
W20260809 16:41:54.367811 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:41:54.498674 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:38647
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:54.584890 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:54.585143 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:54.585183 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:54.586333 11958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:54.586400 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:54.586426 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:54.586455 11958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:54.586485 11958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:54.588025 11958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:54.588308 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:54.588536 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:54.593482 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:54.593499 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:54.593716 11958 server_base.cc:1034] running on GCE node
W20260809 16:41:54.593547 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:54.594074 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:54.594414 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:54.595607 11958 hybrid_clock.cc:648] HybridClock initialized: now 1786293714595590 us; error 48 us; skew 500 ppm
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:54.598168 11958 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:54.598654 11958 webserver.cc:460] Webserver started at http://127.28.62.190:43399/ using document root <none> and password file <none>
I20260809 16:41:54.598903 11958 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:54.598973 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:54.599153 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:54.600215 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "06065a1dfc9f40998f192cd6f4d07549"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b794a8a80de7b8ef2b128978310cdd04"
server_key_iv: "518de45c4e4d41224c7d5c04ba9225f6"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.600582 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "06065a1dfc9f40998f192cd6f4d07549"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b794a8a80de7b8ef2b128978310cdd04"
server_key_iv: "518de45c4e4d41224c7d5c04ba9225f6"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.601948 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:41:54.602736 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:54.602932 11958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:54.603058 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "06065a1dfc9f40998f192cd6f4d07549"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b794a8a80de7b8ef2b128978310cdd04"
server_key_iv: "518de45c4e4d41224c7d5c04ba9225f6"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.603137 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:54.613904 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:54.616189 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:54.616364 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:54.620698 11958 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:38647
I20260809 16:41:54.620757 12033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:38647 every 8 connection(s)
I20260809 16:41:54.621106 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:54.621989 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:54.623701 12034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.624785 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 11958
I20260809 16:41:54.624897 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:41:54.625105 28922 external_mini_cluster.cc:1428] Setting key 9dbe828227cd92c50138a3521b26f72e
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.189@KRBTEST.COM; defaulting to no policy
I20260809 16:41:54.640453 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.624459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.28.62.190:38647)
0809 16:41:54.624631 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.624636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.624676 (+ 40us) server_negotiation.cc:407] Connection header received
0809 16:41:54.625422 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.625424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.625441 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.625517 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.626274 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.627363 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.628214 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.628739 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.628849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.631264 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.631274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.631274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.631282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.633287 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.633684 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.633695 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.633696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.633807 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.634168 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.634169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.634170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.634629 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.634756 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.639996 (+ 5240us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.640176 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260809 16:41:54.640457 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.624397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:38647 (local address 127.0.0.1:53932)
0809 16:41:54.624554 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.624597 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.625275 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.625557 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.625559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.625942 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.626169 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.626175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.627453 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.627454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.628129 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.628130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.628826 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.628827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.628899 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.629443 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.629451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.631175 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.633390 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.633397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.633399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.633590 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.633919 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.633920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.633921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.634072 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.639674 (+ 5602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.639686 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.639799 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.639953 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.640253 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":125,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.189@KRBTEST.COM" created.
I20260809 16:41:54.643301 12034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.643744 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.643610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56756)
0809 16:41:54.643664 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.643718 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 16:41:54.643920 11977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
I20260809 16:41:54.644438 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.644317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56768)
0809 16:41:54.644385 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.644417 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 16:41:54.644570 12034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:40677: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
I20260809 16:41:54.659073 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:40677
--webserver_interface=127.28.62.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:41:54.693830 12034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } attempt: 1
I20260809 16:41:54.694355 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.694198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56778)
0809 16:41:54.694276 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.694322 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 16:41:54.694738 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.694627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56780)
0809 16:41:54.694679 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.694713 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 16:41:54.694834 12034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:40677: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
W20260809 16:41:54.743506 12043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:54.743777 12043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:54.743815 12043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:54.744946 12043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:54.745021 12043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:54.745047 12043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:54.745076 12043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:54.745106 12043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:54.746626 12043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:40677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.62.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:54.746924 12043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:54.747182 12043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:54.752066 12048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:54.752077 12051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:54.752077 12049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:54.752456 12043 server_base.cc:1034] running on GCE node
I20260809 16:41:54.752733 12043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:54.753055 12043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:54.754258 12043 hybrid_clock.cc:648] HybridClock initialized: now 1786293714754245 us; error 45 us; skew 500 ppm
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:54.756822 12043 init.cc:377] Logged in from keytab as kudu/127.28.62.189@KRBTEST.COM (short username kudu)
I20260809 16:41:54.757302 12043 webserver.cc:460] Webserver started at http://127.28.62.189:34951/ using document root <none> and password file <none>
I20260809 16:41:54.757508 12043 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:54.757551 12043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:54.757644 12043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:54.758672 12043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data/instance:
uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "6a203c198332995af014c3208388dd1c"
server_key_iv: "218ab4ad1f8241601bf8efb53501469c"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.758972 12043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal/instance:
uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "6a203c198332995af014c3208388dd1c"
server_key_iv: "218ab4ad1f8241601bf8efb53501469c"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.760370 12043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:54.761158 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:54.761368 12043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:54.761461 12043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "6a203c198332995af014c3208388dd1c"
server_key_iv: "218ab4ad1f8241601bf8efb53501469c"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.761518 12043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:54.762984 12034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } attempt: 2
I20260809 16:41:54.763446 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.763312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56794)
0809 16:41:54.763375 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.763408 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 16:41:54.763850 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.763760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56804)
0809 16:41:54.763808 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.763834 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 16:41:54.763965 12034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:40677: Network error: Client connection negotiation failed: client connection to 127.28.62.189:40677: connect: Connection refused (error 111)
I20260809 16:41:54.778535 12043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:54.781118 12043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:54.781289 12043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:54.785537 12043 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.189:40677
I20260809 16:41:54.785601 12118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.189:40677 every 8 connection(s)
I20260809 16:41:54.785989 12043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
I20260809 16:41:54.786937 12119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:54.788615 12119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:54.795224 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12043
I20260809 16:41:54.795357 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-1/wal/instance
I20260809 16:41:54.795596 28922 external_mini_cluster.cc:1428] Setting key 400a1633a918b370da3ee90aa9a2f736
I20260809 16:41:54.798727 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.789371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.28.62.190:38647)
0809 16:41:54.789446 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.789448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.789639 (+ 191us) server_negotiation.cc:407] Connection header received
0809 16:41:54.790250 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.790251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.790268 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.790399 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.791131 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.791645 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.792633 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.792833 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.792885 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.795297 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.795299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.795299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.795308 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.797129 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.797656 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.797657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.797658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.797723 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.798044 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.798045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.798046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.798175 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.798230 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.798588 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.798659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:41:54.799165 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.789314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:38647 (local address 127.0.0.1:53948)
0809 16:41:54.789500 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.789545 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.790135 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.790511 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.790514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.790786 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.790986 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.790990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.791743 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.791744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.792557 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.792559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.792907 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.792908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.793018 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.793565 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.793574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.795191 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.797238 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.797246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.797248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.797560 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.797802 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.797803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.797804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.797970 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.798247 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.798255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.798437 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.798567 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.798763 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260809 16:41:54.799741 12119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.188@KRBTEST.COM; defaulting to no policy
I20260809 16:41:54.807639 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.800043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56812)
0809 16:41:54.800099 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.800111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.800156 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.800443 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.800444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.800613 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.800688 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.800690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.801558 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.801559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.802457 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.802460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.802927 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.802928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.803030 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.803445 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.803451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.804774 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.806606 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.806607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.806607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.806811 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.806995 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.806996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.806996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.807105 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.807351 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.807352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.807429 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.807486 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.807571 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 16:41:54.807714 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.800130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.28.62.189:40677)
0809 16:41:54.800290 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.800295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.800313 (+ 18us) server_negotiation.cc:407] Connection header received
0809 16:41:54.800339 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.800341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.800352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.800407 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.800778 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.801462 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.802545 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.802851 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.802904 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.804861 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.804871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.804871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.804878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.806515 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.806878 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.806880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.806881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.806933 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.807150 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.807150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.807150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.807298 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.807362 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.807507 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.807619 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.188@KRBTEST.COM" created.
I20260809 16:41:54.808429 12119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.808917 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.808725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52230)
0809 16:41:54.808845 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.808896 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260809 16:41:54.809106 12060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
I20260809 16:41:54.809567 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.809473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52238)
0809 16:41:54.809522 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.809549 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 16:41:54.809697 12119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:37405: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
I20260809 16:41:54.826836 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:37405
--webserver_interface=127.28.62.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:41:54.855170 12034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } attempt: 3
I20260809 16:41:54.858953 12119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } attempt: 1
I20260809 16:41:54.859541 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.859385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52254)
0809 16:41:54.859452 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.859494 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 16:41:54.859933 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.859812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52270)
0809 16:41:54.859867 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.859901 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 16:41:54.860174 12119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:37405: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:41:54.863713 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.855770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.28.62.189:40677)
0809 16:41:54.855843 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.855845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.855858 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:41:54.855935 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.855936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.855955 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.856044 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.856434 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.856999 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.857924 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.858166 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.858256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.860854 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.860855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.860855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.860864 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.862385 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.862766 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.862767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.862768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.862829 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.863073 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.863074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.863075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.863197 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.863285 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.863437 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.863522 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:41:54.863718 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.855655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56822)
0809 16:41:54.855740 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.855763 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.855854 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.856050 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.856052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.856257 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.856327 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.856329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.857143 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.857145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.857789 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.857792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.858279 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.858280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.858366 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.858922 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.858929 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.860749 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.862494 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.862495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.862496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.862676 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.862900 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.862901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.862901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.862963 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.863270 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.863272 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.863322 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.863412 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.863533 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 16:41:54.864401 12034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.864773 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.864683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52274)
0809 16:41:54.864724 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.864754 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260809 16:41:54.865061 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.864998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52286)
0809 16:41:54.865028 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.865048 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260809 16:41:54.865163 12034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:37405: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
I20260809 16:41:54.896346 12034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } attempt: 1
I20260809 16:41:54.896981 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.896832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52290)
0809 16:41:54.896908 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.896951 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 16:41:54.897369 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.897265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52300)
0809 16:41:54.897317 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.897348 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 16:41:54.897464 12034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:37405: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
W20260809 16:41:54.915511 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:54.915732 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:54.915755 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:54.916916 12127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:54.916961 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:54.916975 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:54.916988 12127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:54.917001 12127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:54.918556 12127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:38647,127.28.62.189:40677,127.28.62.188:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.62.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:54.918809 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:54.919111 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:54.924495 12136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:54.924620 12133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:54.924506 12134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:54.924785 12127 server_base.cc:1034] running on GCE node
I20260809 16:41:54.925256 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:54.925597 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:54.926800 12127 hybrid_clock.cc:648] HybridClock initialized: now 1786293714926771 us; error 41 us; skew 500 ppm
I20260809 16:41:54.928370 12119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } attempt: 2
I20260809 16:41:54.928821 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.928696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52314)
0809 16:41:54.928747 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.928790 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 16:41:54.929271 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.929137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52320)
0809 16:41:54.929204 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.929239 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 16:41:54.929395 12119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:37405: Network error: Client connection negotiation failed: client connection to 127.28.62.188:37405: connect: Connection refused (error 111)
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:54.929877 12127 init.cc:377] Logged in from keytab as kudu/127.28.62.188@KRBTEST.COM (short username kudu)
I20260809 16:41:54.930312 12127 webserver.cc:460] Webserver started at http://127.28.62.188:41081/ using document root <none> and password file <none>
I20260809 16:41:54.930594 12127 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:54.930645 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:54.930807 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:54.932009 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data/instance:
uuid: "ccc16a27e83e47d58cc5ccce21f65ef9"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b6c46c956de7c5ac9cce938e49d3ef41"
server_key_iv: "ac46cd7bee7973abb851a4543b769d0d"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.932411 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal/instance:
uuid: "ccc16a27e83e47d58cc5ccce21f65ef9"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b6c46c956de7c5ac9cce938e49d3ef41"
server_key_iv: "ac46cd7bee7973abb851a4543b769d0d"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.933804 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:41:54.934623 12143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:54.934872 12127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:54.934949 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "ccc16a27e83e47d58cc5ccce21f65ef9"
format_stamp: "Formatted at 2026-08-09 16:41:54 on dist-test-slave-wl2h"
server_key: "b6c46c956de7c5ac9cce938e49d3ef41"
server_key_iv: "ac46cd7bee7973abb851a4543b769d0d"
server_key_version: "encryptionkey@0"
I20260809 16:41:54.935070 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:54.944986 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:54.946943 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:54.947198 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:54.951192 12127 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.188:37405
I20260809 16:41:54.951280 12203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.188:37405 every 8 connection(s)
I20260809 16:41:54.951619 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
I20260809 16:41:54.952545 12204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:54.952978 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12127
I20260809 16:41:54.953087 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786293555794774-28922-0/minicluster-data/master-2/wal/instance
I20260809 16:41:54.953317 28922 external_mini_cluster.cc:1428] Setting key 9cee46bf47cdef86b6e4b9a463f9c56b
I20260809 16:41:54.954670 12204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:41:54.964304 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.955529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.28.62.190:38647)
0809 16:41:54.955588 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.955590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.955766 (+ 176us) server_negotiation.cc:407] Connection header received
0809 16:41:54.955816 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.955817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.955827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.955892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.956485 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.956944 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.958118 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.958490 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.958570 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.960941 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.960942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.960943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.960952 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.962558 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.963253 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.963254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.963254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.963311 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.963627 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.963628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.963628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.963754 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.963823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.964139 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.964228 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 16:41:54.964562 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.955427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:38647 (local address 127.0.0.1:53954)
0809 16:41:54.955654 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.955695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.955762 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.955918 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.955920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.956269 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.956370 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.956375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.957041 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.957042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.958018 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.958021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.958592 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.958593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.958675 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.959198 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.959208 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.960847 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.962676 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.962685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.962688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.963172 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.963388 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.963389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.963389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.963534 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.963832 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.963838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.963996 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.964093 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.964278 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
I20260809 16:41:54.965164 12204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.965721 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.953894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.28.62.188:37405)
0809 16:41:54.954127 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.954135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.954162 (+ 27us) server_negotiation.cc:407] Connection header received
0809 16:41:54.954567 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.954573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.954619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.954745 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.955567 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.956774 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.957832 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.958313 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.958436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.961524 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.961532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.961533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.961540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.963779 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.964256 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.964260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.964260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.964325 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.964586 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.964587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.964588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.964837 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.965053 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.965180 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.965281 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:54.972635 12034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } attempt: 2
I20260809 16:41:54.973521 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.965552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.28.62.189:40677)
0809 16:41:54.965602 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.965604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.965641 (+ 37us) server_negotiation.cc:407] Connection header received
0809 16:41:54.965693 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.965700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.965713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.965785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.966145 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.966872 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.967734 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.968057 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.968127 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.970249 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.970250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.970250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.970259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.972238 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.972681 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.972683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.972683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.972749 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.973005 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.973006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.973007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.973126 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.973202 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.973359 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.973431 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:41:54.973521 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.965485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:40677 (local address 127.0.0.1:56834)
0809 16:41:54.965597 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.965607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.965648 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.965804 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.965805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.966016 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.966056 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.966058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.966953 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.966954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.967652 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.967654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.968131 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.968131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.968197 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.968603 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.968608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.970148 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.972353 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.972354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.972354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.972588 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.972837 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.972838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.972838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.972920 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.973208 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.973209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.973250 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.973333 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.973429 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 16:41:54.974207 12204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:41:54.975274 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.966646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.28.62.190:38647)
0809 16:41:54.966686 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.966688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.966697 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:41:54.966729 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.966730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.966739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.966793 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.967389 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.968006 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.968946 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.969193 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.969269 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.971749 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.971750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.971750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.971757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.973637 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.974055 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.974056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.974057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.974114 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.974309 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.974309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.974310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.974428 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.974655 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.975133 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.975211 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
Aug 09 16:41:54 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:41:54.981463 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.972901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52346)
0809 16:41:54.972957 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.972969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.973009 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.973227 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.973228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.973475 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.973522 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.973524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.974552 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.974553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.975368 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.975371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.975955 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.975956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.976011 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.976429 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.976433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.978176 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.980318 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.980319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.980319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.980491 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.980780 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.980781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.980781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.980848 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.981132 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.981133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.981181 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.981249 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.981351 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 16:41:54.981784 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.972960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.28.62.188:37405)
0809 16:41:54.973086 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.973088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.973097 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:41:54.973118 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.973119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.973128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.973300 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.973625 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.974440 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.975470 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.975838 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.975907 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.978281 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.978282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.978283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.978289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.980172 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.980638 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.980639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.980640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.980704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.980935 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.980936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.980936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.981057 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.981140 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.981611 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.981728 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 16:41:54.983599 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.974470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52354)
0809 16:41:54.974621 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:54.974630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:54.974666 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:54.974828 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:54.974828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:54.974971 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:54.975001 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.975002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.976303 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.976304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.977069 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:54.977072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.977557 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.977557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:54.977605 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.978008 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:54.978013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:54.979494 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:54.982443 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.982444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.982444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.982642 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.982867 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:54.982868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:54.982868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:54.982934 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:54.983300 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:54.983301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:54.983347 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 16:41:54.983417 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 16:41:54.983506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260809 16:41:54.983883 12034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: Bootstrap starting.
I20260809 16:41:54.983937 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.974496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.28.62.188:37405)
0809 16:41:54.974684 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.974686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.974710 (+ 24us) server_negotiation.cc:407] Connection header received
0809 16:41:54.974729 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.974730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.974741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.974813 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.975107 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.976210 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.977152 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.977456 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.977513 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.980277 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.980278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.980278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.980284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.982353 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.982722 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.982723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.982723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.982786 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.983009 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.983051 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.983052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.983215 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.983296 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.983711 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.983801 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260809 16:41:54.984416 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.976242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.28.62.189:40677)
0809 16:41:54.976281 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.976283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.976292 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:41:54.976341 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.976342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.976352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.976413 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.976938 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.977429 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.978217 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.978428 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.978506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.981093 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.981094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.981095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.981102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.983074 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.983439 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.983440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.983440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.983504 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:54.983733 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:54.983734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:54.983734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:54.983871 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:54.984066 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:54.984297 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 16:41:54.984357 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:41:54.984712 12034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:54.985008 12034 log.cc:824] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:54.986030 12034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: No bootstrap required, opened a new log
I20260809 16:41:54.986747 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:41:54.986796 12204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: Bootstrap starting.
I20260809 16:41:54.987393 12204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:54.987602 12204 log.cc:824] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:54.988384 12204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: No bootstrap required, opened a new log
I20260809 16:41:54.988252 12034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:54.988452 12034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:54.988498 12034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06065a1dfc9f40998f192cd6f4d07549, State: Initialized, Role: FOLLOWER
I20260809 16:41:54.988636 12034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:54.988888 12218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:54.988950 12034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:54.989034 12218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:54.990638 12204 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:54.990813 12204 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:54.990856 12204 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc16a27e83e47d58cc5ccce21f65ef9, State: Initialized, Role: FOLLOWER
I20260809 16:41:54.990988 12204 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
W20260809 16:41:54.991235 12232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:41:54.991328 12232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:41:54.991343 12204 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:54.991935 11958 master_runner.cc:186] Error getting master registration for 127.28.62.189:40677: OK, instance_id { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" instance_seqno: 1786293714784298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:41:54.996536 11958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 16:41:54.997831 12249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:41:54.997887 12249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:41:54.998430 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:54.998860 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:55.001308 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.988648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.28.62.188:37405)
0809 16:41:54.988696 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.988697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.993362 (+ 4665us) server_negotiation.cc:407] Connection header received
0809 16:41:54.993385 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.993386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.993396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.993440 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.994139 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.994868 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.996020 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.996330 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.996390 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:54.997941 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:54.997943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:54.997944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:54.997950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:54.999753 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.000153 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.000154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.000154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.000211 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.000393 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.000394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.000394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.000515 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:55.000632 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:55.001113 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 16:41:55.001204 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:41:54.997438 12127 master_runner.cc:186] Error getting master registration for 127.28.62.189:40677: OK, instance_id { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" instance_seqno: 1786293714784298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:41:55.003310 12127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 16:41:55.005465 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.993064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.28.62.189:40677)
0809 16:41:54.995154 (+ 2090us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.995156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.996696 (+ 1540us) server_negotiation.cc:407] Connection header received
0809 16:41:54.996807 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.996809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.996821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.996856 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.998365 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.999102 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.000124 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:55.000436 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.000478 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:55.002240 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:55.002241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:55.002242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:55.002249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:55.004079 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.004489 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.004490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.004490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.004550 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.004785 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.004786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.004786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.004923 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:55.005024 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:55.005258 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 16:41:55.005346 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065}
I20260809 16:41:55.006062 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:54.988354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.28.62.190:38647)
0809 16:41:54.988392 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:54.988394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:54.993189 (+ 4795us) server_negotiation.cc:407] Connection header received
0809 16:41:54.993223 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:54.993224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:54.993235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:54.993304 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:54.994079 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.994820 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.995918 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:54.996203 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:54.996284 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:55.002750 (+ 6466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:55.002751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:55.002752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:55.002759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:55.004717 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.005124 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.005125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.005125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.005201 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.005419 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.005420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.005420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.005582 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:55.005683 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:55.005865 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 16:41:55.005947 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:41:55.020545 12119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } attempt: 3
Aug 09 16:41:55 dist-test-slave-wl2h krb5kdc[11944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:41:55.028525 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:55.021270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.28.62.188:37405)
0809 16:41:55.021355 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:55.021357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:55.021364 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:41:55.021384 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:55.021384 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:55.021393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:55.021444 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:55.021844 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:55.022333 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.023006 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:55.023242 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.023324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:55.025286 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:55.025289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:55.025290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:55.025301 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:55.027324 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.027697 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.027698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.027699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.027770 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:55.027978 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:55.027979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:55.027979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:55.028101 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:55.028182 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:55.028347 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 16:41:55.028429 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 16:41:55.028525 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:55.020918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:37405 (local address 127.0.0.1:52380)
0809 16:41:55.021056 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:55.021070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:55.021149 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:55.021500 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:55.021501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:55.021705 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:55.021751 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:55.021753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:55.022422 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.022423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:55.022915 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:55.022916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:55.023323 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:55.023324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:55.023390 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:55.023870 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:55.023875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:55.025180 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:55.027429 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:55.027430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:55.027431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:55.027621 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:55.027842 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:55.027843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:55.027844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:55.027912 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:55.028175 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:55.028176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:55.028224 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 16:41:55.028306 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 16:41:55.028421 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260809 16:41:55.030246 12119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: Bootstrap starting.
I20260809 16:41:55.030925 12119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:55.031294 12119 log.cc:824] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:55.032191 12119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: No bootstrap required, opened a new log
I20260809 16:41:55.033629 12119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:55.033771 12119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:55.033819 12119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9ca9dcf05c4deaa2ed8b9474e1be43, State: Initialized, Role: FOLLOWER
I20260809 16:41:55.033948 12119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:55.034281 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.034343 12119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:55.034451 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 16:41:55.036958 12265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:41:55.037076 12265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:41:55.039040 12043 master_runner.cc:263] No leader master found from master 8f9ca9dcf05c4deaa2ed8b9474e1be43
I20260809 16:41:55.039090 12043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 16:41:55.117163 11981 tablet.cc:2369] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:41:55.197865 12149 tablet.cc:2369] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:41:55.282177 12065 tablet.cc:2369] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:41:55.359225 12218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:41:55.359400 12218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:55.359704 12218 leader_election.cc:288] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9ca9dcf05c4deaa2ed8b9474e1be43 (127.28.62.189:40677), ccc16a27e83e47d58cc5ccce21f65ef9 (127.28.62.188:37405)
I20260809 16:41:55.360070 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06065a1dfc9f40998f192cd6f4d07549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" is_pre_election: true
I20260809 16:41:55.360221 12173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06065a1dfc9f40998f192cd6f4d07549 in term 0.
I20260809 16:41:55.360427 11976 leader_election.cc:302] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06065a1dfc9f40998f192cd6f4d07549, ccc16a27e83e47d58cc5ccce21f65ef9; no voters:
I20260809 16:41:55.360559 12218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:41:55.360615 12218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:41:55.360668 12218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:55.361284 12092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06065a1dfc9f40998f192cd6f4d07549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" is_pre_election: true
I20260809 16:41:55.361435 12092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06065a1dfc9f40998f192cd6f4d07549 in term 0.
I20260809 16:41:55.361721 12218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:55.361881 12218 leader_election.cc:288] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [CANDIDATE]: Term 1 election: Requested vote from peers 8f9ca9dcf05c4deaa2ed8b9474e1be43 (127.28.62.189:40677), ccc16a27e83e47d58cc5ccce21f65ef9 (127.28.62.188:37405)
I20260809 16:41:55.362048 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06065a1dfc9f40998f192cd6f4d07549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9"
I20260809 16:41:55.362120 12092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06065a1dfc9f40998f192cd6f4d07549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43"
I20260809 16:41:55.362149 12173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:55.362206 12092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:55.362905 12092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06065a1dfc9f40998f192cd6f4d07549 in term 1.
I20260809 16:41:55.362908 12173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06065a1dfc9f40998f192cd6f4d07549 in term 1.
I20260809 16:41:55.363097 11977 leader_election.cc:302] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06065a1dfc9f40998f192cd6f4d07549, 8f9ca9dcf05c4deaa2ed8b9474e1be43; no voters:
I20260809 16:41:55.363199 12218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:55.363317 12218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [term 1 LEADER]: Becoming Leader. State: Replica: 06065a1dfc9f40998f192cd6f4d07549, State: Running, Role: LEADER
I20260809 16:41:55.363417 12218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } }
I20260809 16:41:55.363792 12267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06065a1dfc9f40998f192cd6f4d07549. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.363869 12267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:55.364073 12269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:41:55.364270 12269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:55.365558 12092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [term 1 FOLLOWER]: Refusing update from remote peer 06065a1dfc9f40998f192cd6f4d07549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:55.365603 12173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [term 1 FOLLOWER]: Refusing update from remote peer 06065a1dfc9f40998f192cd6f4d07549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:41:55.365880 12218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:55.365983 12267 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:41:55.367002 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06065a1dfc9f40998f192cd6f4d07549. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.367141 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:55.367954 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06065a1dfc9f40998f192cd6f4d07549. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.368054 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:55.368409 12267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.368496 12267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:55.368587 12267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.368646 12267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:55.368741 12269 catalog_manager.cc:1347] Generated new cluster ID: 953c772736154c558c4e6c1d0f2ad2b7
I20260809 16:41:55.368803 12269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:55.368764 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.368842 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:55.368912 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06065a1dfc9f40998f192cd6f4d07549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06065a1dfc9f40998f192cd6f4d07549" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38647 } } peers { permanent_uuid: "8f9ca9dcf05c4deaa2ed8b9474e1be43" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 40677 } } peers { permanent_uuid: "ccc16a27e83e47d58cc5ccce21f65ef9" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 37405 } } }
I20260809 16:41:55.368990 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:41:55.369053 12273 mvcc.cc:205] Tried to move back new op lower bound from 7316659058135228416 to 7316659058129170432. Current Snapshot: MvccSnapshot[applied={T|T < 7316659058135228416}]
I20260809 16:41:55.369105 12271 mvcc.cc:205] Tried to move back new op lower bound from 7316659058135228416 to 7316659058129170432. Current Snapshot: MvccSnapshot[applied={T|T < 7316659058135228416}]
I20260809 16:41:55.376708 12269 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:41:55.377485 12269 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:55.387823 12269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06065a1dfc9f40998f192cd6f4d07549: Generated new TSK 0
I20260809 16:41:55.388065 12269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:41:55.998706 12249 catalog_manager.cc:1259] Loaded cluster ID: 953c772736154c558c4e6c1d0f2ad2b7
I20260809 16:41:55.998793 12249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: loading cluster ID for follower catalog manager: success
I20260809 16:41:55.999677 12249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: acquiring CA information for follower catalog manager: success
I20260809 16:41:56.000007 12249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccc16a27e83e47d58cc5ccce21f65ef9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:41:56.037616 12265 catalog_manager.cc:1259] Loaded cluster ID: 953c772736154c558c4e6c1d0f2ad2b7
I20260809 16:41:56.037711 12265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: loading cluster ID for follower catalog manager: success
I20260809 16:41:56.038581 12265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: acquiring CA information for follower catalog manager: success
I20260809 16:41:56.038934 12265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f9ca9dcf05c4deaa2ed8b9474e1be43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:41:56.459842 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 11958
I20260809 16:41:56.468782 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12043
I20260809 16:41:56.477504 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12127
2026-08-09T16:41:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3249 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:41:56 dist-test-slave-wl2h krb5kdc[12288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:41:56 dist-test-slave-wl2h krb5kdc[12288](info): set up 2 sockets
Aug 09 16:41:56 dist-test-slave-wl2h krb5kdc[12288](info): commencing operation
krb5kdc: starting...
W20260809 16:41:57.620314 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:41:57 dist-test-slave-wl2h krb5kdc[12288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:41:57.746112 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44315
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:57.828958 12301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:57.829216 12301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:57.829258 12301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:57.830456 12301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:57.830529 12301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:57.830556 12301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:57.830585 12301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:57.830607 12301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:57.832237 12301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:57.832527 12301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:57.832773 12301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:57.837888 12306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:57.837934 12307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:57.838200 12301 server_base.cc:1034] running on GCE node
W20260809 16:41:57.837915 12309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:57.838526 12301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:57.838830 12301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:57.839982 12301 hybrid_clock.cc:648] HybridClock initialized: now 1786293717839959 us; error 45 us; skew 500 ppm
Aug 09 16:41:57 dist-test-slave-wl2h krb5kdc[12288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:57.842615 12301 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:57.843128 12301 webserver.cc:460] Webserver started at http://127.28.62.190:42665/ using document root <none> and password file <none>
I20260809 16:41:57.843364 12301 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:57.843410 12301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:57.843550 12301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:57.844667 12301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "ee7e023e186f46fb94a965440a330c48"
format_stamp: "Formatted at 2026-08-09 16:41:57 on dist-test-slave-wl2h"
server_key: "860d526767ff3860eb123406d813c076"
server_key_iv: "ca9531609fcdb49ae3ce442bbff46503"
server_key_version: "encryptionkey@0"
I20260809 16:41:57.845036 12301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "ee7e023e186f46fb94a965440a330c48"
format_stamp: "Formatted at 2026-08-09 16:41:57 on dist-test-slave-wl2h"
server_key: "860d526767ff3860eb123406d813c076"
server_key_iv: "ca9531609fcdb49ae3ce442bbff46503"
server_key_version: "encryptionkey@0"
I20260809 16:41:57.846467 12301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:41:57.847478 12318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:57.847682 12301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:57.847807 12301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "ee7e023e186f46fb94a965440a330c48"
format_stamp: "Formatted at 2026-08-09 16:41:57 on dist-test-slave-wl2h"
server_key: "860d526767ff3860eb123406d813c076"
server_key_iv: "ca9531609fcdb49ae3ce442bbff46503"
server_key_version: "encryptionkey@0"
I20260809 16:41:57.847894 12301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:57.872426 12301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:57.874925 12301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:57.875142 12301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:57.879388 12301 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44315
I20260809 16:41:57.879549 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44315 every 8 connection(s)
I20260809 16:41:57.879841 12301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:57.880805 12378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:57.881660 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12301
I20260809 16:41:57.881800 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:41:57.882009 28922 external_mini_cluster.cc:1428] Setting key ac27784d4dd5124ac1381e2cf239ea5c
I20260809 16:41:57.884151 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48: Bootstrap starting.
I20260809 16:41:57.884910 12378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:57.885244 12378 log.cc:824] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:57.885993 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48: No bootstrap required, opened a new log
I20260809 16:41:57.888119 12378 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } }
I20260809 16:41:57.888267 12378 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:57.888290 12378 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7e023e186f46fb94a965440a330c48, State: Initialized, Role: FOLLOWER
I20260809 16:41:57.888428 12378 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } }
I20260809 16:41:57.888509 12378 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:41:57.888533 12378 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:41:57.888597 12378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 16:41:57 dist-test-slave-wl2h krb5kdc[12288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:57.889133 12378 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } }
I20260809 16:41:57.889209 12378 leader_election.cc:302] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7e023e186f46fb94a965440a330c48; no voters:
I20260809 16:41:57.889439 12378 leader_election.cc:288] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:41:57.889511 12386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:57.889706 12386 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [term 1 LEADER]: Becoming Leader. State: Replica: ee7e023e186f46fb94a965440a330c48, State: Running, Role: LEADER
I20260809 16:41:57.889720 12378 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:57.889853 12386 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } }
I20260809 16:41:57.890252 12387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee7e023e186f46fb94a965440a330c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } } }
I20260809 16:41:57.890282 12388 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7e023e186f46fb94a965440a330c48. Latest consensus state: current_term: 1 leader_uuid: "ee7e023e186f46fb94a965440a330c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7e023e186f46fb94a965440a330c48" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44315 } } }
I20260809 16:41:57.890403 12387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:57.890431 12388 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48 [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:57.890703 12393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:41:57.891253 12393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:57.893025 12393 catalog_manager.cc:1347] Generated new cluster ID: ba0d209c2d2342f6a4084cc153bdd9b2
I20260809 16:41:57.893088 12393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:57.898123 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:57.882570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.28.62.190:44315)
0809 16:41:57.882775 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:57.882785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:57.882818 (+ 33us) server_negotiation.cc:407] Connection header received
0809 16:41:57.883359 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:57.883368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:57.883431 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:57.883581 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:57.884208 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:57.885592 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:57.886435 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:57.886872 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:57.886995 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:57.889724 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:57.889729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:57.889729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:57.889742 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:57.896363 (+ 6621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:57.896853 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:57.896856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:57.896857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:57.896965 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:57.897198 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:57.897199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:57.897199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:57.897495 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:57.897653 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:57.897777 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 16:41:57.897918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260809 16:41:57.899071 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:41:57.900705 12393 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:41:57.901296 12393 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:57.905902 12393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee7e023e186f46fb94a965440a330c48: Generated new TSK 0
I20260809 16:41:57.906136 12393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:41:57.953670 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12301
2026-08-09T16:41:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1475 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:41:58 dist-test-slave-wl2h krb5kdc[12411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:41:58 dist-test-slave-wl2h krb5kdc[12411](info): set up 2 sockets
Aug 09 16:41:58 dist-test-slave-wl2h krb5kdc[12411](info): commencing operation
krb5kdc: starting...
W20260809 16:41:59.087471 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:41:59.217837 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:59.300603 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:59.300808 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:59.300832 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:59.301980 12426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:41:59.302028 12426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:59.302043 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:59.302055 12426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:41:59.302068 12426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:41:59.303699 12426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:59.303961 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:59.304216 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:59.309270 12433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.309233 12432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.309324 12435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:59.309355 12426 server_base.cc:1034] running on GCE node
I20260809 16:41:59.309645 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:59.309947 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:59.311087 12426 hybrid_clock.cc:648] HybridClock initialized: now 1786293719311053 us; error 39 us; skew 500 ppm
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:59.313632 12426 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:41:59.314055 12426 webserver.cc:460] Webserver started at http://127.28.62.190:39227/ using document root <none> and password file <none>
I20260809 16:41:59.314237 12426 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:59.314278 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:59.314366 12426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:59.315496 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "97df25cf9e7b4821aaf0d99e0f64f03c"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "77574c1158982d4adc383052f9a16c90"
server_key_iv: "7af5fa4cef4a8ca510fa1d426f4919e3"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.315817 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "97df25cf9e7b4821aaf0d99e0f64f03c"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "77574c1158982d4adc383052f9a16c90"
server_key_iv: "7af5fa4cef4a8ca510fa1d426f4919e3"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.317080 12426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:41:59.317781 12443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.317974 12426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:41:59.318058 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "97df25cf9e7b4821aaf0d99e0f64f03c"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "77574c1158982d4adc383052f9a16c90"
server_key_iv: "7af5fa4cef4a8ca510fa1d426f4919e3"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.318114 12426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:59.336413 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:59.338435 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:59.338560 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:59.342585 12426 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:34233
I20260809 16:41:59.342648 12503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:34233 every 8 connection(s)
I20260809 16:41:59.343058 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:41:59.343575 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12426
I20260809 16:41:59.343663 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:41:59.343885 28922 external_mini_cluster.cc:1428] Setting key 5d7d663b72b20760f6121a78d38b46ba
I20260809 16:41:59.344352 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:59.347187 12504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap starting.
I20260809 16:41:59.347805 12504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:59.348097 12504 log.cc:824] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:59.348794 12504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: No bootstrap required, opened a new log
I20260809 16:41:59.350556 12504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:41:59.350723 12504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:59.350775 12504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Initialized, Role: FOLLOWER
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:59.350910 12504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:41:59.351016 12504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:41:59.351089 12504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:41:59.351138 12504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:41:59.351840 12504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:41:59.351969 12504 leader_election.cc:302] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97df25cf9e7b4821aaf0d99e0f64f03c; no voters:
I20260809 16:41:59.352154 12504 leader_election.cc:288] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:41:59.352270 12510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:41:59.352439 12504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:41:59.352458 12510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 LEADER]: Becoming Leader. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Running, Role: LEADER
I20260809 16:41:59.352550 12510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:41:59.352911 12511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:41:59.352999 12511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:59.353175 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97df25cf9e7b4821aaf0d99e0f64f03c. Latest consensus state: current_term: 1 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:41:59.353282 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:41:59.353741 12519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:41:59.354295 12519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:41:59.355551 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.344591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.28.62.190:34233)
0809 16:41:59.344780 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.344788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.344813 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:41:59.345746 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.345753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.345803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.345955 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:59.346591 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.347632 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.348493 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.348931 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.349051 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.351516 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:59.351520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:59.351521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:59.351533 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:59.353816 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.354291 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.354294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.354295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.354433 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.354656 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.354657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.354657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.354894 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:59.355081 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.355196 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.355347 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260809 16:41:59.356003 12519 catalog_manager.cc:1347] Generated new cluster ID: e6bd92c49d9742acb21dc906ef71fb17
I20260809 16:41:59.356051 12519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:41:59.369486 12519 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:41:59.370021 12519 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:41:59.376230 12519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Generated new TSK 0
I20260809 16:41:59.376422 12519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:41:59.414471 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:59.506863 12534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:59.507157 12534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:59.507220 12534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:59.508392 12534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:59.508460 12534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:59.508507 12534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:41:59.509982 12534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:59.510252 12534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:59.510514 12534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:59.515748 12545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.515738 12546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.515738 12548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:59.515787 12534 server_base.cc:1034] running on GCE node
I20260809 16:41:59.516203 12534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:59.516450 12534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:59.517598 12534 hybrid_clock.cc:648] HybridClock initialized: now 1786293719517582 us; error 32 us; skew 500 ppm
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:59.520306 12534 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:41:59.520848 12534 webserver.cc:460] Webserver started at http://127.28.62.129:43751/ using document root <none> and password file <none>
I20260809 16:41:59.521081 12534 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:59.521126 12534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:59.521281 12534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:59.522359 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "980fc3172c034393af35b3559f5da60d"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "487dfef2ec23e096cf5421127c3c49da"
server_key_iv: "e66c66f59b993ea06a447295472c042c"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.522728 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "980fc3172c034393af35b3559f5da60d"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "487dfef2ec23e096cf5421127c3c49da"
server_key_iv: "e66c66f59b993ea06a447295472c042c"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.524261 12534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:41:59.525010 12556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.525202 12534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:41:59.525306 12534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "980fc3172c034393af35b3559f5da60d"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "487dfef2ec23e096cf5421127c3c49da"
server_key_iv: "e66c66f59b993ea06a447295472c042c"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.525393 12534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:59.544929 12534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:59.547365 12534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:59.547529 12534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:59.547816 12534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:59.548146 12534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:59.548202 12534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.548236 12534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:59.548267 12534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.555187 12534 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:36119
I20260809 16:41:59.555241 12696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:36119 every 8 connection(s)
I20260809 16:41:59.555614 12534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:41:59.561250 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12534
I20260809 16:41:59.561381 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:41:59.561585 28922 external_mini_cluster.cc:1428] Setting key 6257d4d8c609cabce57e0b38561663f0
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:59.565274 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.556409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:58795 (local address 127.28.62.190:34233)
0809 16:41:59.556492 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.556494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.556706 (+ 212us) server_negotiation.cc:407] Connection header received
0809 16:41:59.557206 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.557207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.557216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.557287 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:59.558209 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.558706 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.559778 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.559983 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.560039 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.562304 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:59.562305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:59.562306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:59.562313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:59.563764 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.564274 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.564275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.564275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.564331 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.564618 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.564618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.564618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.564722 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:59.564772 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.565095 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.565211 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 16:41:59.565519 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.556365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:58795)
0809 16:41:59.556550 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.556592 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.557116 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.557341 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.557344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.557936 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:59.558125 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.558129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.558787 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.558788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.559704 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.559706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.560058 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.560059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.560158 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.560612 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:59.560620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:59.562214 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:59.563859 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.563867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.563869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.564190 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.564403 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.564404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.564405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.564551 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.564785 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:59.564792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:59.564950 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.565076 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.565253 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260809 16:41:59.566110 12697 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:41:59.566212 12697 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:59.566411 12697 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:41:59.566975 12463 ts_manager.cc:194] Registered new tserver with Master: 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119)
I20260809 16:41:59.567623 12463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:58795
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:41:59.590314 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:59.674106 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:59.674376 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:59.674415 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:59.675586 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:59.675666 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:59.675719 12703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:41:59.677222 12703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:59.677501 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:59.677750 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:59.682881 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:59.682998 12703 server_base.cc:1034] running on GCE node
W20260809 16:41:59.682881 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.683094 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:59.683404 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:59.683651 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:59.684827 12703 hybrid_clock.cc:648] HybridClock initialized: now 1786293719684813 us; error 28 us; skew 500 ppm
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:59.687331 12703 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:41:59.687793 12703 webserver.cc:460] Webserver started at http://127.28.62.130:38753/ using document root <none> and password file <none>
I20260809 16:41:59.688030 12703 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:59.688107 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:59.688266 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:59.689304 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "348d311a0eb240e088052198d42db1a8"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "9d63f946998a11f2818f733b91cd0360"
server_key_iv: "cf3e56483381cfe7e13ee6606b939bc4"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.689685 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "348d311a0eb240e088052198d42db1a8"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "9d63f946998a11f2818f733b91cd0360"
server_key_iv: "cf3e56483381cfe7e13ee6606b939bc4"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.691509 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 16:41:59.692600 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.693101 12703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:41:59.693199 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "348d311a0eb240e088052198d42db1a8"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "9d63f946998a11f2818f733b91cd0360"
server_key_iv: "cf3e56483381cfe7e13ee6606b939bc4"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.693269 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:59.725786 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:59.727870 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:59.728044 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:59.728377 12703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:59.728710 12703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:59.728789 12703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.728828 12703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:59.728860 12703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.734660 12703 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:44311
I20260809 16:41:59.734720 12850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:44311 every 8 connection(s)
I20260809 16:41:59.735088 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:41:59.735910 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12703
I20260809 16:41:59.736017 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:41:59.736215 28922 external_mini_cluster.cc:1428] Setting key b749d36cb3a03bd8aba55911bbe7294a
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:41:59.745078 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.735898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:59143 (local address 127.28.62.190:34233)
0809 16:41:59.735972 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.735974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.736176 (+ 202us) server_negotiation.cc:407] Connection header received
0809 16:41:59.736794 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.736795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.736807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.736870 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:59.737840 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.738372 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.739401 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.739621 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.739713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.741979 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:59.741980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:59.741981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:59.741989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:59.743598 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.744047 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.744048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.744049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.744115 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.744398 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.744399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.744399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.744520 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:59.744594 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.744894 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.744981 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:41:59.745301 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.735846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:59143)
0809 16:41:59.736023 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.736067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.736692 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.736949 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.736953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.737535 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:59.737732 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.737737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.738474 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.738475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.739302 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.739304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.739722 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.739722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.739860 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.740430 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:59.740438 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:59.741878 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:59.743723 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.743731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.743733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.743963 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.744208 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.744209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.744209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.744319 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.744600 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:59.744607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:59.744752 (+ 145us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.744860 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.745041 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260809 16:41:59.745852 12851 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:41:59.745959 12851 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:59.746136 12851 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:41:59.746572 12463 ts_manager.cc:194] Registered new tserver with Master: 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311)
I20260809 16:41:59.746917 12463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:59143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:41:59.764155 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:41:59.849447 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:41:59.849690 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:41:59.849716 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:41:59.850860 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:41:59.850903 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:41:59.850955 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:41:59.852501 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:41:59.852744 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:41:59.853016 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:41:59.858105 12863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.858160 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:41:59.858127 12864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:41:59.858350 12858 server_base.cc:1034] running on GCE node
I20260809 16:41:59.858551 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:41:59.858799 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:41:59.859970 12858 hybrid_clock.cc:648] HybridClock initialized: now 1786293719859950 us; error 30 us; skew 500 ppm
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:41:59.862448 12858 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:41:59.862879 12858 webserver.cc:460] Webserver started at http://127.28.62.131:38741/ using document root <none> and password file <none>
I20260809 16:41:59.863190 12858 fs_manager.cc:359] Metadata directory not provided
I20260809 16:41:59.863267 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:41:59.863427 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:41:59.864451 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "85d18748790542e48242ae50e9e1a166"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "4a2eb76111337efc7f05b7d99d8c0072"
server_key_iv: "5500ca2767a2d61830ba62a67d9c3874"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.864817 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "85d18748790542e48242ae50e9e1a166"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "4a2eb76111337efc7f05b7d99d8c0072"
server_key_iv: "5500ca2767a2d61830ba62a67d9c3874"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.866160 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:41:59.866916 12875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.867148 12858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:41:59.867262 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "85d18748790542e48242ae50e9e1a166"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "4a2eb76111337efc7f05b7d99d8c0072"
server_key_iv: "5500ca2767a2d61830ba62a67d9c3874"
server_key_version: "encryptionkey@0"
I20260809 16:41:59.867354 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:41:59.890637 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:41:59.892900 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:41:59.893080 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:41:59.893342 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:41:59.893693 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:41:59.893757 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.893805 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:41:59.893833 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:41:59.900395 12858 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:40149
I20260809 16:41:59.900450 13005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:40149 every 8 connection(s)
I20260809 16:41:59.900801 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:41:59 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:41:59.910321 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.901638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:57091 (local address 127.28.62.190:34233)
0809 16:41:59.901715 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.901718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.901893 (+ 175us) server_negotiation.cc:407] Connection header received
0809 16:41:59.902476 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.902478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.902491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.902552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:59.903663 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.904211 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.905293 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.905508 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.905617 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.907826 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:59.907828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:59.907828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:59.907835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:59.909147 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.909535 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.909535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.909536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.909587 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.909806 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.909807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.909807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.909911 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:59.909962 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.910191 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.910253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:41:59.910451 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 12858
I20260809 16:41:59.910542 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:41:59.910619 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.901559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:57091)
0809 16:41:59.901740 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.901789 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.902356 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.902624 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.902628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.903342 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:41:59.903542 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.903546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.904313 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.904314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.905191 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.905193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.905643 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.905644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.905773 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.906289 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:41:59.906297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:41:59.907737 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:41:59.909245 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.909252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.909253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.909466 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.909649 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:41:59.909649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:41:59.909649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:41:59.909750 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:41:59.909967 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:41:59.909971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:41:59.910081 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.910174 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.910365 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260809 16:41:59.910741 28922 external_mini_cluster.cc:1428] Setting key 60049d4b3b1954d6552f9df3b7a62a58
I20260809 16:41:59.911106 13006 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:41:59.911233 13006 heartbeater.cc:464] Registering TS with master...
I20260809 16:41:59.911435 13006 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:41:59.911830 12463 ts_manager.cc:194] Registered new tserver with Master: 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
I20260809 16:41:59.912274 12463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:57091
I20260809 16:41:59.923007 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:41:59.931303 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.924589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.28.62.190:34233)
0809 16:41:59.924634 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.924638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.924786 (+ 148us) server_negotiation.cc:407] Connection header received
0809 16:41:59.924861 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.924862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.924870 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.924953 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:41:59.925570 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.926059 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.926817 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.927073 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.927161 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.928483 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:41:59.928484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:41:59.928485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:41:59.928493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:41:59.930000 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.930385 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.930386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.930386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.930448 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:41:59.930663 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:41:59.930664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:41:59.930664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:41:59.930821 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:41:59.930929 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.931104 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.931191 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:41:59.932945 12463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:41:59.934561 12463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:41:59.941354 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44311 (local address 127.0.0.1:53196)
0809 16:41:59.937569 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.937578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.937616 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.937879 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.937880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.937881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:59.937925 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.937926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.939352 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.939353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.940343 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.940346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.941045 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.941133 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941148 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.941219 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.941274 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260809 16:41:59.941365 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.28.62.130:44311)
0809 16:41:59.937613 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.937617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.937632 (+ 15us) server_negotiation.cc:407] Connection header received
0809 16:41:59.937679 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.937681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.937698 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.937795 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:59.938003 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.939208 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.940436 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.940957 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941047 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941109 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.941240 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.941283 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260809 16:41:59.941501 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:40149 (local address 127.0.0.1:38908)
0809 16:41:59.937150 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.937191 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.937260 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.937823 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.937825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.937827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:59.937888 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.937889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.939320 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.939322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.940419 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.940421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.941278 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.941335 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941339 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.941371 (+ 32us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.941416 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":24,"thread_start_us":67,"threads_started":1}
I20260809 16:41:59.941623 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.28.62.129:36119)
0809 16:41:59.937585 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.937592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.937602 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:41:59.937657 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.937660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.937676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.937759 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:59.938113 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.938808 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.940327 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.941088 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941224 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.941464 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.941513 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260809 16:41:59.941885 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.28.62.131:40149)
0809 16:41:59.937623 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 16:41:59.937627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:41:59.937637 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:41:59.937680 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:41:59.937681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:41:59.937698 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:41:59.937971 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:41:59.938068 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.939214 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.940485 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.941187 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941678 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941750 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:41:59.941775 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 16:41:59.941820 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":110,"threads_started":1}
I20260809 16:41:59.942094 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:41:59.937106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36119 (local address 127.0.0.1:37264)
0809 16:41:59.937426 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 16:41:59.937434 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:41:59.937493 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:41:59.937771 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:41:59.937774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:41:59.937776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:41:59.938035 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.938038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.938910 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.938911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.940233 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:41:59.940236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:41:59.941262 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:41:59.941264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:41:59.941339 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:41:59.941342 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:41:59.941387 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 16:41:59.941424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260809 16:41:59.942660 12774 tablet_service.cc:1467] Processing CreateTablet for tablet e0129e4fd86c401392e7ea168b766768 (DEFAULT_TABLE table=test-table [id=67ae8e9a19d34381b933df073adec677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:59.942667 12927 tablet_service.cc:1467] Processing CreateTablet for tablet e0129e4fd86c401392e7ea168b766768 (DEFAULT_TABLE table=test-table [id=67ae8e9a19d34381b933df073adec677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:59.942813 12611 tablet_service.cc:1467] Processing CreateTablet for tablet e0129e4fd86c401392e7ea168b766768 (DEFAULT_TABLE table=test-table [id=67ae8e9a19d34381b933df073adec677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:41:59.943063 12774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0129e4fd86c401392e7ea168b766768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:59.943074 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0129e4fd86c401392e7ea168b766768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:59.943208 12611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0129e4fd86c401392e7ea168b766768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:41:59.945904 13026 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Bootstrap starting.
I20260809 16:41:59.946358 13029 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Bootstrap starting.
I20260809 16:41:59.946758 13026 tablet_bootstrap.cc:654] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:59.947043 13029 tablet_bootstrap.cc:654] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:59.947189 13030 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Bootstrap starting.
I20260809 16:41:59.947451 13026 log.cc:824] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:59.947650 13029 log.cc:824] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:59.947866 13030 tablet_bootstrap.cc:654] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Neither blocks nor log segments found. Creating new log.
I20260809 16:41:59.948256 13026 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: No bootstrap required, opened a new log
I20260809 16:41:59.948342 13026 ts_tablet_manager.cc:1397] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260809 16:41:59.948423 13030 log.cc:824] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Log is configured to *not* fsync() on all Append() calls
I20260809 16:41:59.949927 13026 raft_consensus.cc:348] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.950095 13026 raft_consensus.cc:374] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:59.950151 13026 raft_consensus.cc:729] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980fc3172c034393af35b3559f5da60d, State: Initialized, Role: FOLLOWER
I20260809 16:41:59.950276 13026 consensus_queue.cc:260] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.950534 13026 ts_tablet_manager.cc:1428] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:41:59.950590 12697 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:41:59.951192 13029 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: No bootstrap required, opened a new log
I20260809 16:41:59.951284 13029 ts_tablet_manager.cc:1397] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 16:41:59.951462 13030 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: No bootstrap required, opened a new log
I20260809 16:41:59.951534 13030 ts_tablet_manager.cc:1397] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 16:41:59.953166 13029 raft_consensus.cc:348] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.953372 13029 raft_consensus.cc:374] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:59.953418 13029 raft_consensus.cc:729] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348d311a0eb240e088052198d42db1a8, State: Initialized, Role: FOLLOWER
I20260809 16:41:59.953536 13029 consensus_queue.cc:260] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.953510 13030 raft_consensus.cc:348] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.953673 13030 raft_consensus.cc:374] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:41:59.953715 13030 raft_consensus.cc:729] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d18748790542e48242ae50e9e1a166, State: Initialized, Role: FOLLOWER
I20260809 16:41:59.953874 12851 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:41:59.953873 13030 consensus_queue.cc:260] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:41:59.953879 13029 ts_tablet_manager.cc:1428] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:41:59.954190 13006 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:41:59.954200 13030 ts_tablet_manager.cc:1428] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260809 16:41:59.985723 12854 tablet.cc:2369] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:42:00.056412 12698 tablet.cc:2369] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:42:00.147249 13034 raft_consensus.cc:482] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:00.147405 13034 raft_consensus.cc:504] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.147830 13034 leader_election.cc:288] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149), 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311)
I20260809 16:42:00.150949 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.148252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:49683 (local address 127.28.62.130:44311)
0809 16:42:00.148312 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.148314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.148441 (+ 127us) server_negotiation.cc:407] Connection header received
0809 16:42:00.148520 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.148521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.148530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.148609 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.148788 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.149251 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150191 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.150569 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150667 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.150677 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.150835 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.150874 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:42:00.150975 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.148079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44311 (local address 127.28.62.129:49683)
0809 16:42:00.148367 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.148379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.148459 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.148614 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.148615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.148616 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.148709 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.148710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.149339 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.149340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.150085 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.150087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.150667 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.150751 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.150761 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.150827 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.150868 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":247,"threads_started":1}
I20260809 16:42:00.151086 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.148124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:51939 (local address 127.28.62.131:40149)
0809 16:42:00.148177 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.148179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.148243 (+ 64us) server_negotiation.cc:407] Connection header received
0809 16:42:00.148353 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.148354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.148364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.148436 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.148610 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.149097 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150089 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.150473 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150740 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.150752 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.150923 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.150964 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:42:00.151314 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.148008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:40149 (local address 127.28.62.129:51939)
0809 16:42:00.148155 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.148169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.148276 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.148440 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.148441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.148443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.148523 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.148525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.149184 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.149185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.149986 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.149988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.150733 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.150735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.150817 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.150821 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.150913 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.150955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
W20260809 16:42:00.151430 13008 tablet.cc:2369] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:42:00.151403 12796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0129e4fd86c401392e7ea168b766768" candidate_uuid: "980fc3172c034393af35b3559f5da60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348d311a0eb240e088052198d42db1a8" is_pre_election: true
I20260809 16:42:00.151572 12796 raft_consensus.cc:2454] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980fc3172c034393af35b3559f5da60d in term 0.
I20260809 16:42:00.151818 12559 leader_election.cc:302] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348d311a0eb240e088052198d42db1a8, 980fc3172c034393af35b3559f5da60d; no voters:
I20260809 16:42:00.151877 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0129e4fd86c401392e7ea168b766768" candidate_uuid: "980fc3172c034393af35b3559f5da60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d18748790542e48242ae50e9e1a166" is_pre_election: true
I20260809 16:42:00.152063 12951 raft_consensus.cc:2454] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 980fc3172c034393af35b3559f5da60d in term 0.
I20260809 16:42:00.152217 13034 raft_consensus.cc:2781] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:42:00.152304 13034 raft_consensus.cc:482] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:42:00.152334 13034 raft_consensus.cc:3037] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.153259 13034 raft_consensus.cc:504] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.153412 13034 leader_election.cc:288] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 1 election: Requested vote from peers 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149), 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311)
I20260809 16:42:00.153594 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0129e4fd86c401392e7ea168b766768" candidate_uuid: "980fc3172c034393af35b3559f5da60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d18748790542e48242ae50e9e1a166"
I20260809 16:42:00.153594 12796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0129e4fd86c401392e7ea168b766768" candidate_uuid: "980fc3172c034393af35b3559f5da60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348d311a0eb240e088052198d42db1a8"
I20260809 16:42:00.153680 12951 raft_consensus.cc:3037] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.153681 12796 raft_consensus.cc:3037] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.154569 12796 raft_consensus.cc:2454] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980fc3172c034393af35b3559f5da60d in term 1.
I20260809 16:42:00.154570 12951 raft_consensus.cc:2454] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 980fc3172c034393af35b3559f5da60d in term 1.
I20260809 16:42:00.154799 12559 leader_election.cc:302] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348d311a0eb240e088052198d42db1a8, 980fc3172c034393af35b3559f5da60d; no voters:
I20260809 16:42:00.154953 13034 raft_consensus.cc:2781] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:42:00.155195 13034 raft_consensus.cc:686] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 LEADER]: Becoming Leader. State: Replica: 980fc3172c034393af35b3559f5da60d, State: Running, Role: LEADER
I20260809 16:42:00.155334 13034 consensus_queue.cc:237] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.156391 12463 catalog_manager.cc:5495] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d reported cstate change: term changed from 0 to 1, leader changed from <none> to 980fc3172c034393af35b3559f5da60d (127.28.62.129). New cstate: current_term: 1 leader_uuid: "980fc3172c034393af35b3559f5da60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } health_report { overall_health: HEALTHY } } }
I20260809 16:42:00.168634 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.165937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.28.62.129:36119)
0809 16:42:00.165998 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.166000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.166008 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:42:00.166038 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.166038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.166053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.166160 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:00.166327 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.166793 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.167826 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.168018 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.168107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.168271 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:00.168373 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:42:00.168437 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.168553 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.168574 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 16:42:00.170670 12951 raft_consensus.cc:1261] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Refusing update from remote peer 980fc3172c034393af35b3559f5da60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:00.170670 12796 raft_consensus.cc:1261] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Refusing update from remote peer 980fc3172c034393af35b3559f5da60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:00.171046 13034 consensus_queue.cc:1035] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:00.171217 13040 consensus_queue.cc:1035] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:00.174094 13044 mvcc.cc:205] Tried to move back new op lower bound from 7316659077815959552 to 7316659077757612032. Current Snapshot: MvccSnapshot[applied={T|T < 7316659077815959552}]
I20260809 16:42:00.176275 12463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:00.176366 12463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:00.177181 12463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Sending DeleteTablet for 3 replicas of tablet e0129e4fd86c401392e7ea168b766768
I20260809 16:42:00.177618 12774 tablet_service.cc:1514] Processing DeleteTablet for tablet e0129e4fd86c401392e7ea168b766768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:53196
I20260809 16:42:00.177639 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet e0129e4fd86c401392e7ea168b766768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:38908
I20260809 16:42:00.177852 13054 tablet_replica.cc:330] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: stopping tablet replica
I20260809 16:42:00.177867 12611 tablet_service.cc:1514] Processing DeleteTablet for tablet e0129e4fd86c401392e7ea168b766768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37264
I20260809 16:42:00.177958 13054 raft_consensus.cc:2229] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:00.178078 13054 raft_consensus.cc:2258] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.178103 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12534
I20260809 16:42:00.178257 13056 tablet_replica.cc:330] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: stopping tablet replica
I20260809 16:42:00.178351 13056 raft_consensus.cc:2229] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:00.178436 13056 raft_consensus.cc:2258] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.181703 13054 ts_tablet_manager.cc:1905] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.183585 13056 ts_tablet_manager.cc:1905] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.183916 13054 ts_tablet_manager.cc:1918] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:00.183995 13054 log.cc:1196] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/e0129e4fd86c401392e7ea168b766768
I20260809 16:42:00.184338 13054 ts_tablet_manager.cc:1939] T e0129e4fd86c401392e7ea168b766768 P 348d311a0eb240e088052198d42db1a8: Deleting consensus metadata
I20260809 16:42:00.184894 12448 catalog_manager.cc:4841] TS 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): tablet e0129e4fd86c401392e7ea168b766768 (table test-table [id=67ae8e9a19d34381b933df073adec677]) successfully deleted
I20260809 16:42:00.185313 13056 ts_tablet_manager.cc:1918] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:00.185379 13056 log.cc:1196] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/e0129e4fd86c401392e7ea168b766768
I20260809 16:42:00.185580 13056 ts_tablet_manager.cc:1939] T e0129e4fd86c401392e7ea168b766768 P 85d18748790542e48242ae50e9e1a166: Deleting consensus metadata
I20260809 16:42:00.186189 12449 catalog_manager.cc:4841] TS 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): tablet e0129e4fd86c401392e7ea168b766768 (table test-table [id=67ae8e9a19d34381b933df073adec677]) successfully deleted
W20260809 16:42:00.189744 12444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:36119 (error 108)
I20260809 16:42:00.189972 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12703
I20260809 16:42:00.191450 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.190205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36119 (local address 127.0.0.1:37272)
0809 16:42:00.191345 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.191402 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:36119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1113}
W20260809 16:42:00.191635 12444 catalog_manager.cc:4584] TS 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0129e4fd86c401392e7ea168b766768: Network error: Client connection negotiation failed: client connection to 127.28.62.129:36119: connect: Connection refused (error 111)
I20260809 16:42:00.197830 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12858
I20260809 16:42:00.205974 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 12426
I20260809 16:42:00.215118 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--webserver_interface=127.28.62.190
--webserver_port=39227
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:00.301540 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:00.301754 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:00.301776 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:00.302942 13059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:42:00.302999 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:00.303040 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:00.303066 13059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:42:00.303131 13059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:42:00.304689 13059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:00.304951 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:00.305212 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:00.315280 13065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.315277 13066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.315277 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:00.315745 13059 server_base.cc:1034] running on GCE node
I20260809 16:42:00.316030 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:00.316363 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:00.317538 13059 hybrid_clock.cc:648] HybridClock initialized: now 1786293720317520 us; error 43 us; skew 500 ppm
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:00.320297 13059 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:42:00.320837 13059 webserver.cc:460] Webserver started at http://127.28.62.190:39227/ using document root <none> and password file <none>
I20260809 16:42:00.321089 13059 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:00.321156 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:00.322643 13059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.323668 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.323905 13059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:42:00.324021 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "97df25cf9e7b4821aaf0d99e0f64f03c"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "77574c1158982d4adc383052f9a16c90"
server_key_iv: "7af5fa4cef4a8ca510fa1d426f4919e3"
server_key_version: "encryptionkey@0"
I20260809 16:42:00.324528 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:00.352932 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:00.355195 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:00.355356 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:00.360229 13059 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:34233
I20260809 16:42:00.360330 13142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:34233 every 8 connection(s)
I20260809 16:42:00.360754 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:42:00.361260 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13059
I20260809 16:42:00.361620 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:36119
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=43751
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:00.363317 13144 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:42:00.364046 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap starting.
I20260809 16:42:00.373135 13144 log.cc:824] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:00.375408 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:00.375742 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap complete.
I20260809 16:42:00.378324 13144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:00.378556 13144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Initialized, Role: FOLLOWER
I20260809 16:42:00.378726 13144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:00.378799 13144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:42:00.378872 13144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:42:00.378922 13144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:42:00.379987 13144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:00.380071 13144 leader_election.cc:302] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97df25cf9e7b4821aaf0d99e0f64f03c; no voters:
I20260809 16:42:00.380292 13144 leader_election.cc:288] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 16:42:00.380329 13150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 16:42:00.380441 13150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 LEADER]: Becoming Leader. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Running, Role: LEADER
I20260809 16:42:00.380540 13150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:00.380576 13144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:42:00.380827 13152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97df25cf9e7b4821aaf0d99e0f64f03c. Latest consensus state: current_term: 2 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:42:00.381048 13152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:00.380990 13151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:42:00.381322 13151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:00.381424 13158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:42:00.382303 13158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:42:00.382423 13158 catalog_manager.cc:1259] Loaded cluster ID: e6bd92c49d9742acb21dc906ef71fb17
I20260809 16:42:00.382675 13158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:42:00.382903 13158 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 16:42:00.385586 13158 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 16:42:00.387691 13158 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:42:00.387899 13158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Loaded TSK: 0
I20260809 16:42:00.388111 13158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 16:42:00.475332 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:00.475612 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:00.475658 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:00.477010 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:00.477092 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:00.477146 13146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:42:00.478813 13146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:36119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=43751
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:00.479146 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:00.479414 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:00.485087 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.485216 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.485400 13182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:00.485451 13146 server_base.cc:1034] running on GCE node
I20260809 16:42:00.485677 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:00.485977 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:00.487138 13146 hybrid_clock.cc:648] HybridClock initialized: now 1786293720487124 us; error 30 us; skew 500 ppm
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:00.490520 13146 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:42:00.491132 13146 webserver.cc:460] Webserver started at http://127.28.62.129:43751/ using document root <none> and password file <none>
I20260809 16:42:00.491456 13146 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:00.491549 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:00.493410 13146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:42:00.494256 13190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.494472 13146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.494572 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "980fc3172c034393af35b3559f5da60d"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "487dfef2ec23e096cf5421127c3c49da"
server_key_iv: "e66c66f59b993ea06a447295472c042c"
server_key_version: "encryptionkey@0"
I20260809 16:42:00.495041 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:00.509912 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:00.512578 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:00.512769 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:00.513013 13146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:00.513499 13197 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 16:42:00.515683 13146 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 16:42:00.515789 13146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 16:42:00.515856 13146 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 16:42:00.516526 13146 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 16:42:00.516587 13146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:00.516655 13197 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Bootstrap starting.
I20260809 16:42:00.524109 13146 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:36119
I20260809 16:42:00.524150 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:36119 every 8 connection(s)
I20260809 16:42:00.524555 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:42:00.528306 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13146
I20260809 16:42:00.528612 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:44311
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=38753
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:00.528605 13197 log.cc:824] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Log is configured to *not* fsync() on all Append() calls
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:00.534572 13197 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:00.534970 13197 tablet_bootstrap.cc:492] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Bootstrap complete.
I20260809 16:42:00.536445 13197 ts_tablet_manager.cc:1397] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.014s
I20260809 16:42:00.537812 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.525295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:60039)
0809 16:42:00.525627 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.525670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.526120 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.526700 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.526712 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.527006 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:00.527260 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.527265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.528874 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.528875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.530168 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.530171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.531325 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.531326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.531685 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.532433 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:00.532442 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:00.533910 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:00.536033 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.536040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.536042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.536281 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.536606 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.536607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.536607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.536716 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.537151 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:00.537158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:00.537294 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.537407 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.537610 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260809 16:42:00.537843 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.525425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:60039 (local address 127.28.62.190:34233)
0809 16:42:00.526313 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.526323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.526365 (+ 42us) server_negotiation.cc:407] Connection header received
0809 16:42:00.526481 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.526485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.526533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.526670 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:00.527402 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.528733 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.530302 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.531169 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.532065 (+ 896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.534066 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:00.534070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:00.534070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:00.534080 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:00.535916 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.536376 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.536380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.536380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.536491 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.536797 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.536798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.536799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.537063 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:00.537172 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.537478 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.537610 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":82,"threads_started":1}
I20260809 16:42:00.537931 13197 raft_consensus.cc:348] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.538303 13197 raft_consensus.cc:729] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980fc3172c034393af35b3559f5da60d, State: Initialized, Role: FOLLOWER
I20260809 16:42:00.538465 13197 consensus_queue.cc:260] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.538808 13197 ts_tablet_manager.cc:1428] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260809 16:42:00.539085 13326 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:42:00.539199 13326 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:00.539392 13326 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:42:00.540131 13095 ts_manager.cc:194] Registered new tserver with Master: 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119)
I20260809 16:42:00.541347 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:60039
I20260809 16:42:00.542207 13326 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:42:00.546466 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.541661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.28.62.129:36119)
0809 16:42:00.543406 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.543410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.543417 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:42:00.543443 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.543445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.543455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.543541 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.543766 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.544412 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.545479 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.545961 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.546064 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.546127 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.546340 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.546384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717,"thread_start_us":97,"threads_started":1}
I20260809 16:42:00.546478 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.540758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36119 (local address 127.0.0.1:37284)
0809 16:42:00.543201 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.543236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.543297 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.543564 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.543567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.543569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.543673 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.543675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.544528 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.544529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.545379 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.545381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.546089 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.546091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.546186 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.546200 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.546320 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.546384 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2403,"thread_start_us":107,"threads_started":1}
I20260809 16:42:00.547232 13245 tablet_service.cc:1514] Processing DeleteTablet for tablet e0129e4fd86c401392e7ea168b766768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260809 16:42:00.547784 13337 tablet_replica.cc:330] stopping tablet replica
I20260809 16:42:00.547921 13337 raft_consensus.cc:2229] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:00.547996 13337 raft_consensus.cc:2258] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.548332 13337 ts_tablet_manager.cc:1905] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.550825 13337 ts_tablet_manager.cc:1918] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:00.550921 13337 log.cc:1196] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/e0129e4fd86c401392e7ea168b766768
I20260809 16:42:00.551190 13337 ts_tablet_manager.cc:1939] T e0129e4fd86c401392e7ea168b766768 P 980fc3172c034393af35b3559f5da60d: Deleting consensus metadata
I20260809 16:42:00.551836 13077 catalog_manager.cc:4841] TS 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119): tablet e0129e4fd86c401392e7ea168b766768 (table test-table [id=67ae8e9a19d34381b933df073adec677]) successfully deleted
W20260809 16:42:00.634791 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:00.635104 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:00.635144 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:00.636299 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:00.636338 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:00.636364 13332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:42:00.637835 13332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:44311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=38753
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:00.638048 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:00.638245 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:00.643513 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.643519 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.643502 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:00.643791 13332 server_base.cc:1034] running on GCE node
I20260809 16:42:00.643987 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:00.644191 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:00.645370 13332 hybrid_clock.cc:648] HybridClock initialized: now 1786293720645357 us; error 40 us; skew 500 ppm
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:00.648111 13332 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:42:00.648550 13332 webserver.cc:460] Webserver started at http://127.28.62.130:38753/ using document root <none> and password file <none>
I20260809 16:42:00.648815 13332 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:00.648877 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:00.650240 13332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.650976 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.651291 13332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.651399 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "348d311a0eb240e088052198d42db1a8"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "9d63f946998a11f2818f733b91cd0360"
server_key_iv: "cf3e56483381cfe7e13ee6606b939bc4"
server_key_version: "encryptionkey@0"
I20260809 16:42:00.651798 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:00.670680 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:00.673166 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:00.673353 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:00.673558 13332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:00.673839 13332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:00.673870 13332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.673893 13332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:00.673906 13332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.680317 13332 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:44311
I20260809 16:42:00.680421 13501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:44311 every 8 connection(s)
I20260809 16:42:00.680781 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:42:00.683506 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13332
I20260809 16:42:00.683782 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:40149
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=38741
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:00.691401 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.681643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:48983 (local address 127.28.62.190:34233)
0809 16:42:00.681695 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.681697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.682001 (+ 304us) server_negotiation.cc:407] Connection header received
0809 16:42:00.682478 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.682479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.682494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.682550 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:00.683570 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.684210 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.685228 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.685457 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.685539 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.688049 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:00.688050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:00.688051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:00.688059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:00.689700 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.690256 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.690257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.690257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.690324 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.690605 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.690607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.690607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.690743 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:00.690832 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.691180 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.691294 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:42:00.691601 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.681683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:48983)
0809 16:42:00.681872 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.681908 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.682401 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.682606 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.682609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.683268 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:00.683461 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.683465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.684311 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.684312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.685130 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.685132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.685545 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.685546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.685688 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.686261 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:00.686269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:00.687940 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:00.689864 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.689871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.689874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.690134 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.690403 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.690404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.690405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.690526 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.690822 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:00.690830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:00.690985 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.691148 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.691329 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260809 16:42:00.692162 13502 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:42:00.692286 13502 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:00.692499 13502 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:42:00.692910 13095 ts_manager.cc:194] Registered new tserver with Master: 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311)
I20260809 16:42:00.693485 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:48983
W20260809 16:42:00.771389 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:00.771596 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:00.771621 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:00.772742 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:00.772786 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:00.772812 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:42:00.774195 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:40149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=38741
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:00.774398 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:00.774583 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:00.779805 13513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:00.779790 13514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:00.780089 13507 server_base.cc:1034] running on GCE node
W20260809 16:42:00.779790 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:00.780383 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:00.780658 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:00.781805 13507 hybrid_clock.cc:648] HybridClock initialized: now 1786293720781784 us; error 58 us; skew 500 ppm
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:00.784495 13507 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:42:00.784970 13507 webserver.cc:460] Webserver started at http://127.28.62.131:38741/ using document root <none> and password file <none>
I20260809 16:42:00.785157 13507 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:00.785210 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:00.786559 13507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.787228 13524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.787446 13507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:00.787521 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "85d18748790542e48242ae50e9e1a166"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "4a2eb76111337efc7f05b7d99d8c0072"
server_key_iv: "5500ca2767a2d61830ba62a67d9c3874"
server_key_version: "encryptionkey@0"
I20260809 16:42:00.787858 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:00.794797 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:00.797302 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:00.797426 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:00.797614 13507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:00.797951 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:00.797988 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.798012 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:00.798055 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.804330 13507 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:40149
I20260809 16:42:00.804373 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:40149 every 8 connection(s)
I20260809 16:42:00.804733 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:42:00.810199 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13507
Aug 09 16:42:00 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:00.815435 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.805516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:49457 (local address 127.28.62.190:34233)
0809 16:42:00.805587 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.805589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.805782 (+ 193us) server_negotiation.cc:407] Connection header received
0809 16:42:00.806292 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.806294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.806306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.806374 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:00.807409 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.807909 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.808941 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.809136 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.809196 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.812060 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:00.812062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:00.812062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:00.812070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:00.813779 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.814330 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.814331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.814332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.814397 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.814677 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.814677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.814678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.814796 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:00.814874 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.815254 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.815366 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:42:00.815693 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.805491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:49457)
0809 16:42:00.805649 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.805692 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.806180 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.806426 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.806430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.807099 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:00.807321 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.807325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.808004 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.808005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.808847 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.808849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.809236 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.809237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.809345 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.809939 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:00.809948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:00.811933 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:00.813883 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.813892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.813894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.814243 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.814468 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:00.814469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:00.814469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:00.814597 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:00.814866 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:00.814872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:00.815073 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.815217 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.815404 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260809 16:42:00.816277 13657 heartbeater.cc:347] Connected to a master server at 127.28.62.190:34233
I20260809 16:42:00.816385 13657 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:00.816602 13657 heartbeater.cc:511] Master 127.28.62.190:34233 requested a full tablet report, sending...
I20260809 16:42:00.816999 13095 ts_manager.cc:194] Registered new tserver with Master: 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
I20260809 16:42:00.817389 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:49457
I20260809 16:42:00.818257 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.810710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.28.62.190:34233)
0809 16:42:00.810855 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.810857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.810894 (+ 37us) server_negotiation.cc:407] Connection header received
0809 16:42:00.810983 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.810984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.810993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.811107 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:00.811895 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.812481 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.813668 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.813881 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.813936 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.815174 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:00.815176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:00.815176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:00.815184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:00.817026 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.817377 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.817378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.817379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.817458 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:00.817726 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:00.817726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:00.817727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:00.817840 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:00.818038 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.818075 (+ 37us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.818155 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260809 16:42:00.819005 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:42:00.822399 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.819434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.28.62.190:34233)
0809 16:42:00.819468 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.819470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.820185 (+ 715us) server_negotiation.cc:407] Connection header received
0809 16:42:00.820221 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.820222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.820248 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.820306 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:00.820382 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.820821 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.821648 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.821847 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.821901 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.822067 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:00.822183 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:42:00.822250 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.822335 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.822354 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 16:42:00.822736 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:42:00.824679 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:42:00.827154 13245 tablet_service.cc:1467] Processing CreateTablet for tablet f6209d12c75f4086af7e7eac95af8246 (DEFAULT_TABLE table=test-table [id=33efa4cc7b3742e18d3ce6082f90f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:00.827386 13245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6209d12c75f4086af7e7eac95af8246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:00.829170 13197 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Bootstrap starting.
I20260809 16:42:00.829509 13197 tablet_bootstrap.cc:654] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:00.830152 13197 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: No bootstrap required, opened a new log
I20260809 16:42:00.830194 13197 ts_tablet_manager.cc:1397] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:00.830307 13197 raft_consensus.cc:348] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.830386 13197 raft_consensus.cc:374] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:00.830439 13197 raft_consensus.cc:729] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980fc3172c034393af35b3559f5da60d, State: Initialized, Role: FOLLOWER
I20260809 16:42:00.830482 13197 consensus_queue.cc:260] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.830560 13197 ts_tablet_manager.cc:1428] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:00.831591 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.826787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44311 (local address 127.0.0.1:53208)
0809 16:42:00.826853 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.826865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.826907 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.827646 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.827647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.827649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.827686 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.827688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.829163 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.829164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.830227 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.830231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.831332 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.831333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.831420 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.831424 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.831489 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.831540 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 16:42:00.831648 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.827247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.28.62.130:44311)
0809 16:42:00.827427 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.827432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.827444 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:42:00.827495 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.827497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.827515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.827724 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.827805 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.829033 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.830333 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.831196 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.831333 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.831404 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.831506 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.831551 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260809 16:42:00.832355 13417 tablet_service.cc:1467] Processing CreateTablet for tablet f6209d12c75f4086af7e7eac95af8246 (DEFAULT_TABLE table=test-table [id=33efa4cc7b3742e18d3ce6082f90f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:00.832721 13417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6209d12c75f4086af7e7eac95af8246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:00.833382 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.826787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:40149 (local address 127.0.0.1:38916)
0809 16:42:00.829252 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.829263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.829323 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.829536 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.829537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.829538 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.829587 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.829589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.830885 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.830887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.832108 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.832111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.833099 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.833100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.833180 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.833184 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.833238 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.833281 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2448,"mutex_wait_us":28,"thread_start_us":59,"threads_started":1}
I20260809 16:42:00.833673 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.826968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.28.62.131:40149)
0809 16:42:00.827193 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.827198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.829321 (+ 2123us) server_negotiation.cc:407] Connection header received
0809 16:42:00.829398 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.829401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.829417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.829537 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.829659 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.830757 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.832199 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.832937 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.833009 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.833061 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.833547 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.833595 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260809 16:42:00.834336 13581 tablet_service.cc:1467] Processing CreateTablet for tablet f6209d12c75f4086af7e7eac95af8246 (DEFAULT_TABLE table=test-table [id=33efa4cc7b3742e18d3ce6082f90f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:00.834694 13581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6209d12c75f4086af7e7eac95af8246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:00.836615 13672 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Bootstrap starting.
I20260809 16:42:00.837210 13672 tablet_bootstrap.cc:654] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:00.837668 13672 log.cc:824] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:00.838354 13674 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Bootstrap starting.
I20260809 16:42:00.838502 13672 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: No bootstrap required, opened a new log
I20260809 16:42:00.838572 13672 ts_tablet_manager.cc:1397] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:42:00.839047 13674 tablet_bootstrap.cc:654] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:00.839597 13674 log.cc:824] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:00.840549 13674 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: No bootstrap required, opened a new log
I20260809 16:42:00.840626 13674 ts_tablet_manager.cc:1397] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:42:00.840794 13672 raft_consensus.cc:348] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.840960 13672 raft_consensus.cc:374] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:00.840994 13672 raft_consensus.cc:729] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348d311a0eb240e088052198d42db1a8, State: Initialized, Role: FOLLOWER
I20260809 16:42:00.841152 13672 consensus_queue.cc:260] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.841446 13672 ts_tablet_manager.cc:1428] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:42:00.841575 13502 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:42:00.842535 13674 raft_consensus.cc:348] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.842655 13674 raft_consensus.cc:374] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:00.842676 13674 raft_consensus.cc:729] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d18748790542e48242ae50e9e1a166, State: Initialized, Role: FOLLOWER
I20260809 16:42:00.842787 13674 consensus_queue.cc:260] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.843096 13674 ts_tablet_manager.cc:1428] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 16:42:00.843266 13657 heartbeater.cc:503] Master 127.28.62.190:34233 was elected leader, sending a full tablet report...
I20260809 16:42:00.873772 13676 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:00.873966 13676 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.874397 13676 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149), 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119)
I20260809 16:42:00.878016 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.874737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:37485 (local address 127.28.62.129:36119)
0809 16:42:00.874796 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.874798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.874807 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:42:00.874842 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.874843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.874854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.874937 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.875226 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.876031 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877221 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.877583 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877644 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.877654 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.877876 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.877932 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:42:00.878055 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.874652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:40149 (local address 127.28.62.130:41471)
0809 16:42:00.874874 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.874885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.874946 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.875125 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.875126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.875128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.875203 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.875205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.876142 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.876143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.877037 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.877039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.877755 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.877812 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.877816 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.877894 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.877936 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":55,"thread_start_us":126,"threads_started":1}
I20260809 16:42:00.878132 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.874737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:41471 (local address 127.28.62.131:40149)
0809 16:42:00.874785 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.874788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.874964 (+ 176us) server_negotiation.cc:407] Connection header received
0809 16:42:00.874999 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.875000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.875033 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.875104 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:00.875311 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.876031 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877158 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.877646 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.877739 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.877921 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.877970 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:42:00.878114 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.874652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36119 (local address 127.28.62.130:37485)
0809 16:42:00.874721 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.874735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:00.874782 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:00.874979 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:00.874980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:00.874982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:00.875134 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.875137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.876168 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.876169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.877069 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:00.877072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.877672 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.877673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:00.877789 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.877793 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:00.877856 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 16:42:00.877928 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":34}
I20260809 16:42:00.878479 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6209d12c75f4086af7e7eac95af8246" candidate_uuid: "348d311a0eb240e088052198d42db1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d18748790542e48242ae50e9e1a166" is_pre_election: true
I20260809 16:42:00.878674 13604 raft_consensus.cc:2454] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348d311a0eb240e088052198d42db1a8 in term 0.
I20260809 16:42:00.878732 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6209d12c75f4086af7e7eac95af8246" candidate_uuid: "348d311a0eb240e088052198d42db1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980fc3172c034393af35b3559f5da60d" is_pre_election: true
I20260809 16:42:00.878912 13271 raft_consensus.cc:2454] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 348d311a0eb240e088052198d42db1a8 in term 0.
I20260809 16:42:00.878921 13360 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166; no voters:
I20260809 16:42:00.879066 13676 raft_consensus.cc:2781] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:42:00.879160 13676 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:42:00.879199 13676 raft_consensus.cc:3037] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.879935 13676 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.880106 13676 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [CANDIDATE]: Term 1 election: Requested vote from peers 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149), 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119)
I20260809 16:42:00.880261 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6209d12c75f4086af7e7eac95af8246" candidate_uuid: "348d311a0eb240e088052198d42db1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d18748790542e48242ae50e9e1a166"
I20260809 16:42:00.880324 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6209d12c75f4086af7e7eac95af8246" candidate_uuid: "348d311a0eb240e088052198d42db1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "980fc3172c034393af35b3559f5da60d"
I20260809 16:42:00.880345 13604 raft_consensus.cc:3037] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.880415 13271 raft_consensus.cc:3037] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:00.881350 13271 raft_consensus.cc:2454] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348d311a0eb240e088052198d42db1a8 in term 1.
I20260809 16:42:00.881361 13604 raft_consensus.cc:2454] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 348d311a0eb240e088052198d42db1a8 in term 1.
I20260809 16:42:00.881551 13360 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166; no voters:
I20260809 16:42:00.881695 13676 raft_consensus.cc:2781] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:42:00.881867 13676 raft_consensus.cc:686] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 348d311a0eb240e088052198d42db1a8, State: Running, Role: LEADER
I20260809 16:42:00.881980 13676 consensus_queue.cc:237] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:00.882848 13094 catalog_manager.cc:5495] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 348d311a0eb240e088052198d42db1a8 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "348d311a0eb240e088052198d42db1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20260809 16:42:00.892520 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.889926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.28.62.130:44311)
0809 16:42:00.889991 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:00.889993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:00.890003 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:42:00.890036 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:00.890037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:00.890105 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:00.890188 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:00.890320 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.890909 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.891692 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:00.891891 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:00.891968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:00.892150 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:00.892258 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:42:00.892359 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:00.892378 (+ 19us) server_negotiation.cc:299] Negotiation successful
0809 16:42:00.892402 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:42:00.894420 13604 raft_consensus.cc:1261] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Refusing update from remote peer 348d311a0eb240e088052198d42db1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:00.894420 13271 raft_consensus.cc:1261] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Refusing update from remote peer 348d311a0eb240e088052198d42db1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:00.894842 13676 consensus_queue.cc:1035] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:00.895077 13679 consensus_queue.cc:1035] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:00.897624 13685 mvcc.cc:205] Tried to move back new op lower bound from 7316659080780554240 to 7316659080733360128. Current Snapshot: MvccSnapshot[applied={T|T < 7316659080780554240}]
I20260809 16:42:00.897850 13684 mvcc.cc:205] Tried to move back new op lower bound from 7316659080780554240 to 7316659080733360128. Current Snapshot: MvccSnapshot[applied={T|T < 7316659080780554240}]
I20260809 16:42:00.898180 13683 mvcc.cc:205] Tried to move back new op lower bound from 7316659080780554240 to 7316659080733360128. Current Snapshot: MvccSnapshot[applied={T|T < 7316659080780554240 or (T in {7316659080780554240})}]
I20260809 16:42:00.899899 13095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:00.900017 13095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:00.900910 13095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Sending DeleteTablet for 3 replicas of tablet f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:00.901286 13245 tablet_service.cc:1514] Processing DeleteTablet for tablet f6209d12c75f4086af7e7eac95af8246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260809 16:42:00.901291 13417 tablet_service.cc:1514] Processing DeleteTablet for tablet f6209d12c75f4086af7e7eac95af8246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:53208
I20260809 16:42:00.901310 13581 tablet_service.cc:1514] Processing DeleteTablet for tablet f6209d12c75f4086af7e7eac95af8246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:00 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:38916
I20260809 16:42:00.901393 13337 tablet_replica.cc:330] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: stopping tablet replica
I20260809 16:42:00.901446 13337 raft_consensus.cc:2229] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:00.901499 13337 raft_consensus.cc:2258] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.901721 13697 tablet_replica.cc:330] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: stopping tablet replica
I20260809 16:42:00.901742 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13146
I20260809 16:42:00.901861 13696 tablet_replica.cc:330] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: stopping tablet replica
I20260809 16:42:00.901922 13337 ts_tablet_manager.cc:1905] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.902055 13697 raft_consensus.cc:2229] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:00.901973 13696 raft_consensus.cc:2229] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:42:00.902262 13696 raft_consensus.cc:2258] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.902426 13697 raft_consensus.cc:2258] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:00.904242 13696 ts_tablet_manager.cc:1905] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.904621 13697 ts_tablet_manager.cc:1905] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:00.906399 13696 ts_tablet_manager.cc:1918] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:00.906486 13696 log.cc:1196] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:00.906819 13696 ts_tablet_manager.cc:1939] T f6209d12c75f4086af7e7eac95af8246 P 348d311a0eb240e088052198d42db1a8: Deleting consensus metadata
I20260809 16:42:00.907083 13697 ts_tablet_manager.cc:1918] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:00.907196 13697 log.cc:1196] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:00.907354 13079 catalog_manager.cc:4841] TS 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): tablet f6209d12c75f4086af7e7eac95af8246 (table test-table [id=33efa4cc7b3742e18d3ce6082f90f208]) successfully deleted
I20260809 16:42:00.907539 13697 ts_tablet_manager.cc:1939] T f6209d12c75f4086af7e7eac95af8246 P 85d18748790542e48242ae50e9e1a166: Deleting consensus metadata
I20260809 16:42:00.907958 13080 catalog_manager.cc:4841] TS 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): tablet f6209d12c75f4086af7e7eac95af8246 (table test-table [id=33efa4cc7b3742e18d3ce6082f90f208]) successfully deleted
W20260809 16:42:00.913753 13077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:36119 (error 108)
I20260809 16:42:00.914217 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13332
I20260809 16:42:00.914342 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:00.914180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:36119 (local address 127.0.0.1:37288)
0809 16:42:00.914253 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:00.914301 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:36119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 16:42:00.917255 13077 catalog_manager.cc:4584] TS 980fc3172c034393af35b3559f5da60d (127.28.62.129:36119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6209d12c75f4086af7e7eac95af8246: Network error: Client connection negotiation failed: client connection to 127.28.62.129:36119: connect: Connection refused (error 111)
I20260809 16:42:00.929438 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13507
I20260809 16:42:00.940238 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13059
I20260809 16:42:00.951232 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--webserver_interface=127.28.62.190
--webserver_port=39227
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:01.043170 13700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:01.043380 13700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:01.043402 13700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:01.044564 13700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:42:01.044607 13700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:01.044622 13700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:01.044636 13700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:42:01.044692 13700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:42:01.046190 13700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:01.046454 13700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:01.046700 13700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:01.051918 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.052001 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.052075 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:01.052214 13700 server_base.cc:1034] running on GCE node
I20260809 16:42:01.052497 13700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:01.052790 13700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:01.053954 13700 hybrid_clock.cc:648] HybridClock initialized: now 1786293721053941 us; error 45 us; skew 500 ppm
Aug 09 16:42:01 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:01.056607 13700 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:42:01.057089 13700 webserver.cc:460] Webserver started at http://127.28.62.190:39227/ using document root <none> and password file <none>
I20260809 16:42:01.057324 13700 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:01.057397 13700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:01.058794 13700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.059609 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.059796 13700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:01.059896 13700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "97df25cf9e7b4821aaf0d99e0f64f03c"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "77574c1158982d4adc383052f9a16c90"
server_key_iv: "7af5fa4cef4a8ca510fa1d426f4919e3"
server_key_version: "encryptionkey@0"
I20260809 16:42:01.060264 13700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:01.072099 13700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:01.074512 13700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:01.074682 13700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:01.079142 13700 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:34233
I20260809 16:42:01.079181 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:34233 every 8 connection(s)
I20260809 16:42:01.079592 13700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:42:01.081036 13777 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:42:01.081645 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap starting.
I20260809 16:42:01.088243 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13700
I20260809 16:42:01.088575 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:36119
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=43751
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:01.091606 13777 log.cc:824] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:01.094020 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:01.094357 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c: Bootstrap complete.
I20260809 16:42:01.096792 13777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:01.097016 13777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Initialized, Role: FOLLOWER
I20260809 16:42:01.097191 13777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:01.097287 13777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:42:01.097343 13777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:42:01.097419 13777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 2 FOLLOWER]: Advancing to term 3
I20260809 16:42:01.098445 13777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:01.098569 13777 leader_election.cc:302] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97df25cf9e7b4821aaf0d99e0f64f03c; no voters:
I20260809 16:42:01.098764 13777 leader_election.cc:288] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 16:42:01.098869 13782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 16:42:01.099051 13782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [term 3 LEADER]: Becoming Leader. State: Replica: 97df25cf9e7b4821aaf0d99e0f64f03c, State: Running, Role: LEADER
I20260809 16:42:01.099062 13777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:42:01.099205 13782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } }
I20260809 16:42:01.099689 13783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:42:01.099704 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97df25cf9e7b4821aaf0d99e0f64f03c. Latest consensus state: current_term: 3 leader_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97df25cf9e7b4821aaf0d99e0f64f03c" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 34233 } } }
I20260809 16:42:01.099804 13783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:01.099813 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97df25cf9e7b4821aaf0d99e0f64f03c [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:01.100016 13790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:42:01.100793 13790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:42:01.100898 13790 catalog_manager.cc:1259] Loaded cluster ID: e6bd92c49d9742acb21dc906ef71fb17
I20260809 16:42:01.100963 13790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 16:42:01.101441 13790 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff25ae0c9ad google::LogMessage::Fail() at ??:0
@ 0x7ff25ae0f88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff25ae0c4de google::LogMessage::Flush() at ??:0
@ 0x7ff25ae0da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff262d2d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff262d368b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff25ccda0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff25cccf954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff25db336db start_thread at ??:0
@ 0x7ff25990a61f clone at ??:0
W20260809 16:42:01.208506 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:01.208724 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:01.208748 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:01.209915 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:01.209957 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:01.209985 13779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:42:01.211520 13779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:36119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=43751
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:01.211769 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:01.212024 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:01.217159 13822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.217176 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.217175 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:01.217535 13779 server_base.cc:1034] running on GCE node
I20260809 16:42:01.217799 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:01.218047 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:01.219224 13779 hybrid_clock.cc:648] HybridClock initialized: now 1786293721219210 us; error 31 us; skew 500 ppm
Aug 09 16:42:01 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:01.221773 13779 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:42:01.222244 13779 webserver.cc:460] Webserver started at http://127.28.62.129:43751/ using document root <none> and password file <none>
I20260809 16:42:01.222536 13779 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:01.222611 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:01.224134 13779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.224996 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.225204 13779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.225340 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "980fc3172c034393af35b3559f5da60d"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "487dfef2ec23e096cf5421127c3c49da"
server_key_iv: "e66c66f59b993ea06a447295472c042c"
server_key_version: "encryptionkey@0"
I20260809 16:42:01.225695 13779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:01.236912 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:01.239656 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:01.239818 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:01.240091 13779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:01.240706 13844 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 16:42:01.242779 13779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 16:42:01.242861 13779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 16:42:01.242913 13779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 16:42:01.243604 13779 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 16:42:01.243667 13779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:01.243777 13844 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Bootstrap starting.
I20260809 16:42:01.245770 13844 log.cc:824] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:01.248855 13844 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:01.249162 13844 tablet_bootstrap.cc:492] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Bootstrap complete.
I20260809 16:42:01.249320 13844 ts_tablet_manager.cc:1397] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 16:42:01.251106 13779 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:36119
I20260809 16:42:01.251507 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:42:01.251191 13844 raft_consensus.cc:348] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:01.251582 13844 raft_consensus.cc:729] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980fc3172c034393af35b3559f5da60d, State: Initialized, Role: FOLLOWER
I20260809 16:42:01.251766 13844 consensus_queue.cc:260] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:01.251147 13976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:36119 every 8 connection(s)
I20260809 16:42:01.252244 13844 ts_tablet_manager.cc:1428] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:42:01.255841 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13779
I20260809 16:42:01.256158 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:44311
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=38753
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:01.332835 13982 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:42:01.252392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:45275)
0809 16:42:01.252554 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.252589 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:01.252762 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:01.332608 (+ 79846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
W20260809 16:42:01.333102 13977 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:34233 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:34233: Client connection negotiation failed: client connection to 127.28.62.190:34233: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 16:42:01.333487 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.333361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:60631)
0809 16:42:01.333418 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.333463 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 16:42:01.333757 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.333683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:36851)
0809 16:42:01.333713 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.333734 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260809 16:42:01.333863 13977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:42:01.357035 13983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:01.357255 13983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:01.357280 13983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:01.358446 13983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:01.358491 13983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:01.358518 13983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:42:01.360062 13983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:44311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=38753
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:01.360317 13983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:01.360558 13983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:01.365723 13989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.365725 13988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:01.365860 13983 server_base.cc:1034] running on GCE node
W20260809 16:42:01.365765 13993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:01.366192 13983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:01.366396 13983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:01.367563 13983 hybrid_clock.cc:648] HybridClock initialized: now 1786293721367548 us; error 49 us; skew 500 ppm
Aug 09 16:42:01 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:01.370038 13983 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:42:01.370429 13983 webserver.cc:460] Webserver started at http://127.28.62.130:38753/ using document root <none> and password file <none>
I20260809 16:42:01.370622 13983 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:01.370666 13983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:01.371989 13983 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.372581 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.372778 13983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.372869 13983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "348d311a0eb240e088052198d42db1a8"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "9d63f946998a11f2818f733b91cd0360"
server_key_iv: "cf3e56483381cfe7e13ee6606b939bc4"
server_key_version: "encryptionkey@0"
I20260809 16:42:01.373229 13983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:01.390002 13983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:01.392518 13983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:01.392676 13983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:01.392937 13983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:01.393262 13983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:01.393296 13983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.393319 13983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:01.393364 13983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.399369 13983 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:44311
I20260809 16:42:01.399454 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:44311 every 8 connection(s)
I20260809 16:42:01.399794 13983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:42:01.400813 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.400435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:41461)
0809 16:42:01.400586 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.400667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260809 16:42:01.401230 14135 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:34233 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:34233: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
I20260809 16:42:01.401500 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.401413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:38925)
0809 16:42:01.401456 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.401481 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 16:42:01.401753 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.401690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:33791)
0809 16:42:01.401721 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.401741 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260809 16:42:01.401866 14135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 16:42:01.403231 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 13983
I20260809 16:42:01.403487 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:40149
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:34233
--webserver_port=38741
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:01.486440 14139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:01.486652 14139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:01.486678 14139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:01.487877 14139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:01.487919 14139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:01.487946 14139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:42:01.489372 14139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:40149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=38741
--tserver_master_addrs=127.28.62.190:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:01.489603 14139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:01.489856 14139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:01.494911 14147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.494936 14146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:01.494911 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:01.495425 14139 server_base.cc:1034] running on GCE node
I20260809 16:42:01.495658 14139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:01.495894 14139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:01.497057 14139 hybrid_clock.cc:648] HybridClock initialized: now 1786293721497044 us; error 28 us; skew 500 ppm
Aug 09 16:42:01 dist-test-slave-wl2h krb5kdc[12411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:01.499616 14139 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:42:01.500072 14139 webserver.cc:460] Webserver started at http://127.28.62.131:38741/ using document root <none> and password file <none>
I20260809 16:42:01.500263 14139 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:01.500306 14139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:01.501577 14139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:01.502110 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.502295 14139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.502382 14139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "85d18748790542e48242ae50e9e1a166"
format_stamp: "Formatted at 2026-08-09 16:41:59 on dist-test-slave-wl2h"
server_key: "4a2eb76111337efc7f05b7d99d8c0072"
server_key_iv: "5500ca2767a2d61830ba62a67d9c3874"
server_key_version: "encryptionkey@0"
I20260809 16:42:01.502719 14139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:01.547621 14139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:01.550237 14139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:01.550400 14139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:01.550683 14139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:01.551047 14139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:01.551105 14139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.551141 14139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:01.551168 14139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:01.557337 14139 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:40149
I20260809 16:42:01.557400 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:40149 every 8 connection(s)
I20260809 16:42:01.557819 14139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:42:01.559038 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.558633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:41781)
0809 16:42:01.558795 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.558877 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260809 16:42:01.559470 14296 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:34233 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:34233: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
I20260809 16:42:01.559809 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.559677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:44679)
0809 16:42:01.559730 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.559787 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 16:42:01.559983 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 14139
I20260809 16:42:01.560176 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:01.560068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:39533)
0809 16:42:01.560128 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:01.560161 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 16:42:01.560352 14296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 16:42:02.334811 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.334410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:37057)
0809 16:42:02.334645 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:02.334699 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
I20260809 16:42:02.402720 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.402334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:60513)
0809 16:42:02.402529 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:02.402585 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260809 16:42:02.560933 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.560710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:39413)
0809 16:42:02.560844 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:02.560881 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260809 16:42:02.960592 14304 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:02.960744 14304 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:02.961216 14304 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311), 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
Aug 09 16:42:02 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.130@KRBTEST.COM
Aug 09 16:42:02 dist-test-slave-wl2h krb5kdc[12411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.131@KRBTEST.COM
I20260809 16:42:02.972586 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.961624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:39899 (local address 127.28.62.130:44311)
0809 16:42:02.961807 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:02.961815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:02.961833 (+ 18us) server_negotiation.cc:407] Connection header received
0809 16:42:02.962258 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:02.962260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:02.962301 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:02.962466 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:02.963361 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.964412 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.965545 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.965933 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.966064 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:02.968768 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:02.968778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:02.968779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:02.968789 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:02.970761 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971284 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:02.971287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:02.971288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:02.971387 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971659 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:02.971660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:02.971660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:02.971901 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:02.972004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:02.972351 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 16:42:02.972456 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260809 16:42:02.972659 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.961362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:44311 (local address 127.28.62.129:39899)
0809 16:42:02.961530 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:02.961542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:02.961604 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:02.962501 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:02.962504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:02.962947 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:02.963230 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:02.963237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.964526 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.964527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:02.965414 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:02.965417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.966048 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.966049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:02.966155 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:02.966826 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:02.966830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:02.968623 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:02.970870 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:02.970878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:02.970882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:02.971202 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:02.971460 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:02.971461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:02.971584 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:02.971987 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:02.971995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:02.972180 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 16:42:02.972282 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 16:42:02.972524 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":97,"thread_start_us":97,"threads_started":1}
I20260809 16:42:02.972726 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.961420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:40149 (local address 127.28.62.129:59875)
0809 16:42:02.961563 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:02.961576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:02.961643 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:02.962501 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:02.962510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:02.962946 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:02.963471 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:02.963474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.964638 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.964638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:02.965412 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:02.965414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.966005 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.966006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:02.966163 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:02.966773 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:02.966781 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:02.969133 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:02.970966 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:02.970967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:02.970967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:02.971250 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:02.971518 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:02.971519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:02.971590 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:02.972093 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:02.972094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:02.972162 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 16:42:02.972282 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 16:42:02.972582 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":156,"thread_start_us":37,"threads_started":1}
I20260809 16:42:02.973045 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:02.961556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59875 (local address 127.28.62.131:40149)
0809 16:42:02.961754 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:02.961767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:02.961814 (+ 47us) server_negotiation.cc:407] Connection header received
0809 16:42:02.962258 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:02.962261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:02.962301 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:02.962465 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:02.963582 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.964515 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.965502 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:02.965909 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:02.966016 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:02.969267 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:02.969271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:02.969271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:02.969281 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:02.970871 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971347 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:02.971352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:02.971356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:02.971444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:02.971664 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:02.971665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:02.971665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:02.971968 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:02.972064 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:02.972789 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 16:42:02.972948 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
W20260809 16:42:02.973882 13841 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
W20260809 16:42:02.973980 13840 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:02.974048 13840 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980fc3172c034393af35b3559f5da60d; no voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166
I20260809 16:42:02.974181 14304 raft_consensus.cc:2726] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:42:03.335673 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:03.335480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:48811)
0809 16:42:03.335565 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:03.335611 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 16:42:03.403600 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:03.403310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:37627)
0809 16:42:03.403489 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:03.403547 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260809 16:42:03.561780 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:03.561547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:33495)
0809 16:42:03.561690 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:03.561734 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260809 16:42:04.336630 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:04.336347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:48193)
0809 16:42:04.336517 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:04.336564 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260809 16:42:04.404321 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:04.404103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:41165)
0809 16:42:04.404243 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:04.404279 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260809 16:42:04.562677 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:04.562440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:35015)
0809 16:42:04.562595 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:04.562633 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260809 16:42:04.610862 14317 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:04.610961 14317 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:04.611213 14317 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311), 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
W20260809 16:42:04.611658 13840 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
W20260809 16:42:04.611771 13841 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:04.611852 13841 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980fc3172c034393af35b3559f5da60d; no voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166
I20260809 16:42:04.611976 14317 raft_consensus.cc:2726] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:42:05.337386 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:05.337167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:48567)
0809 16:42:05.337301 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:05.337337 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260809 16:42:05.405148 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:05.404842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:51811)
0809 16:42:05.405054 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:05.405103 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260809 16:42:05.563486 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:05.563239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:43263)
0809 16:42:05.563391 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:05.563443 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260809 16:42:06.338415 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:06.338099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:49207)
0809 16:42:06.338318 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:06.338366 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260809 16:42:06.405896 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:06.405678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:51169)
0809 16:42:06.405809 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:06.405848 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260809 16:42:06.564353 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:06.564093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:60749)
0809 16:42:06.564262 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:06.564305 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260809 16:42:06.770623 14325 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:06.770725 14325 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:06.770926 14325 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311), 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
W20260809 16:42:06.771404 13840 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
W20260809 16:42:06.771508 13841 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:06.771607 13841 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980fc3172c034393af35b3559f5da60d; no voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166
I20260809 16:42:06.771716 14325 raft_consensus.cc:2726] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:42:07.339392 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:07.339014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:33267)
0809 16:42:07.339292 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:07.339340 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":190,"threads_started":1}
I20260809 16:42:07.406527 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:07.406314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:37391)
0809 16:42:07.406447 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:07.406483 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260809 16:42:07.565142 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:07.564863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:53223)
0809 16:42:07.565054 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:07.565093 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260809 16:42:08.340200 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:08.339932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:57989)
0809 16:42:08.340109 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:08.340150 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260809 16:42:08.407318 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:08.407081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:35583)
0809 16:42:08.407226 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:08.407271 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260809 16:42:08.565992 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:08.565749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:40487)
0809 16:42:08.565901 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:08.565944 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260809 16:42:09.340993 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:09.340740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:55503)
0809 16:42:09.340903 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:09.340941 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260809 16:42:09.408039 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:09.407827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:42655)
0809 16:42:09.407955 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:09.407993 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260809 16:42:09.566825 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:09.566595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:34399)
0809 16:42:09.566733 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:09.566776 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260809 16:42:10.341768 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:10.341527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:35441)
0809 16:42:10.341682 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:10.341718 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260809 16:42:10.408867 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:10.408609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:46149)
0809 16:42:10.408774 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:10.408820 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260809 16:42:10.567708 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:10.567424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:50657)
0809 16:42:10.567610 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:10.567657 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260809 16:42:11.154989 14340 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:11.155186 14340 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:11.155398 14340 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311), 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
W20260809 16:42:11.155887 13840 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
W20260809 16:42:11.155970 13841 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:11.156036 13841 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980fc3172c034393af35b3559f5da60d; no voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166
I20260809 16:42:11.156134 14340 raft_consensus.cc:2726] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:42:11.342547 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:11.342311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:42701)
0809 16:42:11.342457 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:11.342497 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260809 16:42:11.409657 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:11.409406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:51733)
0809 16:42:11.409557 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:11.409609 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260809 16:42:11.568675 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:11.568415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:41465)
0809 16:42:11.568583 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:11.568626 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260809 16:42:12.343338 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:12.343108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:54435)
0809 16:42:12.343245 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:12.343286 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260809 16:42:12.410401 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:12.410153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:51467)
0809 16:42:12.410312 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:12.410356 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260809 16:42:12.569617 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:12.569348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:54077)
0809 16:42:12.569516 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:12.569567 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260809 16:42:13.344187 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:13.343915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:54263)
0809 16:42:13.344085 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:13.344135 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260809 16:42:13.411415 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:13.411181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:37275)
0809 16:42:13.411323 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:13.411369 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260809 16:42:13.528294 14351 raft_consensus.cc:482] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:13.528453 14351 raft_consensus.cc:504] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348d311a0eb240e088052198d42db1a8" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 44311 } } peers { permanent_uuid: "85d18748790542e48242ae50e9e1a166" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 40149 } } peers { permanent_uuid: "980fc3172c034393af35b3559f5da60d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 36119 } }
I20260809 16:42:13.528681 14351 leader_election.cc:288] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311), 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149)
W20260809 16:42:13.529057 13840 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348d311a0eb240e088052198d42db1a8 (127.28.62.130:44311): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
W20260809 16:42:13.529162 13841 leader_election.cc:341] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85d18748790542e48242ae50e9e1a166 (127.28.62.131:40149): Not found: Tablet not found: f6209d12c75f4086af7e7eac95af8246
I20260809 16:42:13.529243 13841 leader_election.cc:302] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 980fc3172c034393af35b3559f5da60d; no voters: 348d311a0eb240e088052198d42db1a8, 85d18748790542e48242ae50e9e1a166
I20260809 16:42:13.529313 14351 raft_consensus.cc:2726] T f6209d12c75f4086af7e7eac95af8246 P 980fc3172c034393af35b3559f5da60d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:42:13.570398 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:13.570140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:46283)
0809 16:42:13.570307 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:13.570351 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260809 16:42:14.345172 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:14.344847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:58157)
0809 16:42:14.345053 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:14.345107 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260809 16:42:14.412171 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:14.411885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:59791)
0809 16:42:14.412084 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:14.412123 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":66,"threads_started":1}
I20260809 16:42:14.571360 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:14.571075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:45089)
0809 16:42:14.571228 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:14.571293 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260809 16:42:15.346052 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:15.345821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:38719)
0809 16:42:15.345954 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:15.346004 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260809 16:42:15.412947 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:15.412686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:47161)
0809 16:42:15.412860 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:15.412901 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260809 16:42:15.572144 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:15.571897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:59029)
0809 16:42:15.572038 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:15.572096 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260809 16:42:16.346845 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:16.346620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.129:35243)
0809 16:42:16.346759 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:16.346798 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260809 16:42:16.413771 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:16.413541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.130:46061)
0809 16:42:16.413679 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:16.413724 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260809 16:42:16.565955 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13779
I20260809 16:42:16.572911 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:16.572705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:34233 (local address 127.28.62.131:49425)
0809 16:42:16.572837 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:16.572870 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260809 16:42:16.574195 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 13983
I20260809 16:42:16.581627 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 14139
2026-08-09T16:42:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18629 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:42:16 dist-test-slave-wl2h krb5kdc[14366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:42:16 dist-test-slave-wl2h krb5kdc[14366](info): set up 2 sockets
Aug 09 16:42:16 dist-test-slave-wl2h krb5kdc[14366](info): commencing operation
krb5kdc: starting...
W20260809 16:42:17.721264 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:42:17 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:42:17.854609 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:17.938104 14383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:17.938331 14383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:17.938356 14383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:17.939492 14383 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 16:42:17.939527 14383 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 16:42:17.939540 14383 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 16:42:17.939553 14383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:42:17.939565 14383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:17.939579 14383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:17.939590 14383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:42:17.939602 14383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:42:17.941026 14383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:17.941226 14383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:17.941401 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:42:17.946828 14383 server_base.cc:1034] running on GCE node
W20260809 16:42:17.946764 14388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:17.946774 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:17.946796 14390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:17.947289 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:17.947592 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:17.948750 14383 hybrid_clock.cc:648] HybridClock initialized: now 1786293737948730 us; error 33 us; skew 500 ppm
Aug 09 16:42:17 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:17.951393 14383 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:42:17.951851 14383 webserver.cc:460] Webserver started at http://127.28.62.190:39247/ using document root <none> and password file <none>
I20260809 16:42:17.952092 14383 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:17.952167 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:17.952323 14383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:42:17.953442 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "57be2e0243764193b055b5ad5a7cc7fe"
format_stamp: "Formatted at 2026-08-09 16:42:17 on dist-test-slave-wl2h"
server_key: "abbcf768a852e43bd72955ec98a67972"
server_key_iv: "40a3da58cc92965503b515b1b7335b80"
server_key_version: "encryptionkey@0"
I20260809 16:42:17.953815 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "57be2e0243764193b055b5ad5a7cc7fe"
format_stamp: "Formatted at 2026-08-09 16:42:17 on dist-test-slave-wl2h"
server_key: "abbcf768a852e43bd72955ec98a67972"
server_key_iv: "40a3da58cc92965503b515b1b7335b80"
server_key_version: "encryptionkey@0"
I20260809 16:42:17.955266 14383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:42:17.956025 14401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:17.956218 14383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:42:17.956323 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "57be2e0243764193b055b5ad5a7cc7fe"
format_stamp: "Formatted at 2026-08-09 16:42:17 on dist-test-slave-wl2h"
server_key: "abbcf768a852e43bd72955ec98a67972"
server_key_iv: "40a3da58cc92965503b515b1b7335b80"
server_key_version: "encryptionkey@0"
I20260809 16:42:17.956396 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:17.982221 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:17.984488 14383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:17.984676 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:17.989179 14383 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35167
I20260809 16:42:17.989248 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35167 every 8 connection(s)
I20260809 16:42:17.989610 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:42:17.990386 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 14383
I20260809 16:42:17.990494 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:42:17.990612 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:17.990729 28922 external_mini_cluster.cc:1428] Setting key 8196dd428278ce11fd037fc6b28c5358
I20260809 16:42:17.993996 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap starting.
I20260809 16:42:17.994642 14464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:17.994940 14464 log.cc:824] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:17.995641 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: No bootstrap required, opened a new log
Aug 09 16:42:17 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:17.997718 14464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:17.997870 14464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:17.997934 14464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Initialized, Role: FOLLOWER
I20260809 16:42:17.998088 14464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:17.998167 14464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:42:17.998212 14464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:42:17.998262 14464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:17.998914 14464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:17.999063 14464 leader_election.cc:302] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57be2e0243764193b055b5ad5a7cc7fe; no voters:
I20260809 16:42:17.999383 14464 leader_election.cc:288] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:42:17.999444 14471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:42:17.999693 14471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 LEADER]: Becoming Leader. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Running, Role: LEADER
I20260809 16:42:17.999715 14464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:42:17.999825 14471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:18.000267 14472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:18.000422 14472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:18.001425 14484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:42:18.001586 14473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57be2e0243764193b055b5ad5a7cc7fe. Latest consensus state: current_term: 1 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:18.001698 14473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:18.001983 14484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:42:18.002925 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:17.991397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.28.62.190:35167)
0809 16:42:17.991613 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:17.991627 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:17.991653 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:42:17.992212 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:17.992218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:17.992268 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:17.992397 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:17.993197 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:17.994157 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:17.994965 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:17.995435 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:17.995556 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:17.997904 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:17.997909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:17.997909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:17.997922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:18.000590 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.001158 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.001161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.001162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.001279 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.001517 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.001518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.001518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.001856 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:18.002023 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.002572 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.002715 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260809 16:42:18.003963 14484 catalog_manager.cc:1347] Generated new cluster ID: 2227970d8c25468db5777fad11521bad
I20260809 16:42:18.004019 14484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:42:18.014448 14484 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:42:18.014984 14484 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:42:18.021967 14484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Generated new TSK 0
I20260809 16:42:18.022179 14484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:42:18.054244 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:18.138924 14500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:18.139182 14500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:18.139209 14500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:18.140416 14500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:18.140461 14500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:18.140488 14500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:42:18.141940 14500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:18.142196 14500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:18.142441 14500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:18.147493 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:18.147497 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:18.147761 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:18.147763 14500 server_base.cc:1034] running on GCE node
I20260809 16:42:18.148028 14500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:18.148293 14500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:18.149441 14500 hybrid_clock.cc:648] HybridClock initialized: now 1786293738149424 us; error 39 us; skew 500 ppm
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:18.152020 14500 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:42:18.152498 14500 webserver.cc:460] Webserver started at http://127.28.62.129:40413/ using document root <none> and password file <none>
I20260809 16:42:18.152724 14500 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:18.152778 14500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:18.152907 14500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:42:18.153980 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "50aae5b2b806a3c3d49ae5a227866ae2"
server_key_iv: "24781f8a9cee383fc0d61ab596d411b0"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.154340 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "50aae5b2b806a3c3d49ae5a227866ae2"
server_key_iv: "24781f8a9cee383fc0d61ab596d411b0"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.155835 14500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:42:18.156656 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.156881 14500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:18.156985 14500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "50aae5b2b806a3c3d49ae5a227866ae2"
server_key_iv: "24781f8a9cee383fc0d61ab596d411b0"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.157061 14500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:18.175776 14500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:18.178078 14500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:18.178268 14500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:18.178558 14500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:18.178951 14500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:18.179009 14500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.179124 14500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:18.179162 14500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.185912 14500 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:42057
I20260809 16:42:18.185971 14648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:42057 every 8 connection(s)
I20260809 16:42:18.186318 14500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:42:18.190339 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 14500
I20260809 16:42:18.190439 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:42:18.190653 28922 external_mini_cluster.cc:1428] Setting key 7a80cf98922c89e9feb0cf880dac40c8
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:18.196946 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.187108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:43883 (local address 127.28.62.190:35167)
0809 16:42:18.187156 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.187158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.187430 (+ 272us) server_negotiation.cc:407] Connection header received
0809 16:42:18.188000 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.188001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.188015 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.188070 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:18.189092 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.189608 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.190721 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.191090 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.191159 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.193910 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:18.193912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:18.193912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:18.193921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:18.195483 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.195948 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.195949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.195950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.196007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.196265 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.196266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.196266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.196389 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:18.196458 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.196745 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.196850 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:42:18.197147 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.187112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:43883)
0809 16:42:18.187298 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.187336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.187889 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.188138 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.188142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.188777 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:18.188978 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.188982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.189704 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.189705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.190619 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.190622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.191216 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.191217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.191380 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.191980 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:18.191992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:18.193800 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:18.195590 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.195599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.195602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.195858 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.196081 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.196081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.196082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.196193 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.196461 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:18.196469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:18.196612 (+ 143us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.196713 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.196887 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260809 16:42:18.197726 14649 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:18.197844 14649 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:18.198035 14649 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:18.198612 14423 ts_manager.cc:194] Registered new tserver with Master: 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057)
I20260809 16:42:18.199318 14423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:43883
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:42:18.224015 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:18.307945 14654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:18.308166 14654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:18.308192 14654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:18.309257 14654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:18.309293 14654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:18.309317 14654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:42:18.310700 14654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:18.310896 14654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:18.311116 14654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:18.316403 14664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:18.316555 14654 server_base.cc:1034] running on GCE node
W20260809 16:42:18.316413 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:18.316452 14662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:18.316872 14654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:18.317183 14654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:18.318351 14654 hybrid_clock.cc:648] HybridClock initialized: now 1786293738318344 us; error 35 us; skew 500 ppm
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:18.320883 14654 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:42:18.321386 14654 webserver.cc:460] Webserver started at http://127.28.62.130:33651/ using document root <none> and password file <none>
I20260809 16:42:18.321625 14654 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:18.321692 14654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:18.321834 14654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:42:18.322865 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "b510862baef34f859fb6d2f6f463c9f2"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "7be4ca700d1c48d2b7a22c47feebc8cd"
server_key_iv: "970764e26ab99731c0f9393f62769ee5"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.323239 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "b510862baef34f859fb6d2f6f463c9f2"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "7be4ca700d1c48d2b7a22c47feebc8cd"
server_key_iv: "970764e26ab99731c0f9393f62769ee5"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.324551 14654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:18.325249 14671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.325441 14654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:42:18.325538 14654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b510862baef34f859fb6d2f6f463c9f2"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "7be4ca700d1c48d2b7a22c47feebc8cd"
server_key_iv: "970764e26ab99731c0f9393f62769ee5"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.325613 14654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:18.333484 14654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:18.335457 14654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:18.335616 14654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:18.335841 14654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:18.336151 14654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:18.336203 14654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.336246 14654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:18.336272 14654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.342198 14654 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36961
I20260809 16:42:18.342242 14812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36961 every 8 connection(s)
I20260809 16:42:18.342546 14654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:18.349582 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 14654
I20260809 16:42:18.349707 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:42:18.349881 28922 external_mini_cluster.cc:1428] Setting key 51cee05a273662f89d88066dd4c1e2e7
I20260809 16:42:18.352413 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.343427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:60921 (local address 127.28.62.190:35167)
0809 16:42:18.343472 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.343474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.343704 (+ 230us) server_negotiation.cc:407] Connection header received
0809 16:42:18.344337 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.344338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.344350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.344408 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:18.345376 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.345880 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.346864 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.347113 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.347196 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.349388 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:18.349390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:18.349391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:18.349399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:18.350989 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.351442 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.351443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.351444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.351505 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.351750 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.351752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.351752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.351877 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:18.351948 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.352236 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.352320 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:42:18.352763 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.343398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:60921)
0809 16:42:18.343579 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.343615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.344221 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.344473 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.344477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.345054 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:18.345252 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.345257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.345983 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.345984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.346773 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.346775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.347209 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.347210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.347343 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.347903 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:18.347911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:18.349266 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:18.351110 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.351117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.351119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.351350 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.351578 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.351579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.351579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.351684 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.351950 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:18.351959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:18.352101 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.352203 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.352376 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260809 16:42:18.353248 14813 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:18.353361 14813 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:18.353544 14813 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:18.353969 14423 ts_manager.cc:194] Registered new tserver with Master: b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
I20260809 16:42:18.354377 14423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:60921
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:42:18.378269 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:18.461499 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:18.461731 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:18.461771 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:18.462894 14819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:18.462957 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:18.463002 14819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:42:18.464552 14819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:18.464833 14819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:18.465072 14819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:18.470108 14824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:18.470093 14825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:18.470093 14827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:18.470438 14819 server_base.cc:1034] running on GCE node
I20260809 16:42:18.470672 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:18.470873 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:18.472012 14819 hybrid_clock.cc:648] HybridClock initialized: now 1786293738471996 us; error 32 us; skew 500 ppm
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:18.474565 14819 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:42:18.474962 14819 webserver.cc:460] Webserver started at http://127.28.62.131:35683/ using document root <none> and password file <none>
I20260809 16:42:18.475235 14819 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:18.475318 14819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:18.475414 14819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:42:18.476552 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "73761d49041b463aa59438092f4903e9"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "e9e401744360f6eafbeec267875f6567"
server_key_iv: "1622c63315a8eac84e2ae60fec8f6e84"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.476869 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "73761d49041b463aa59438092f4903e9"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "e9e401744360f6eafbeec267875f6567"
server_key_iv: "1622c63315a8eac84e2ae60fec8f6e84"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.478133 14819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:42:18.478854 14836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.479102 14819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:18.479189 14819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "73761d49041b463aa59438092f4903e9"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "e9e401744360f6eafbeec267875f6567"
server_key_iv: "1622c63315a8eac84e2ae60fec8f6e84"
server_key_version: "encryptionkey@0"
I20260809 16:42:18.479244 14819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:18.497602 14819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:18.500007 14819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:18.500128 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:18.500404 14819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:18.500804 14819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:18.500842 14819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.500865 14819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:18.500877 14819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:18.507113 14819 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:32883
I20260809 16:42:18.507169 14971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:32883 every 8 connection(s)
I20260809 16:42:18.507498 14819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:42:18 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:18.513926 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 14819
I20260809 16:42:18.514025 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:42:18.514245 28922 external_mini_cluster.cc:1428] Setting key c3ce2b5e694adcc0d1c4e84dad754f4d
I20260809 16:42:18.517187 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.508234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:38555 (local address 127.28.62.190:35167)
0809 16:42:18.508311 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.508314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.508563 (+ 249us) server_negotiation.cc:407] Connection header received
0809 16:42:18.509154 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.509155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.509166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.509226 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:18.510182 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.510686 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.511678 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.511877 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.511942 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.514205 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:18.514206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:18.514207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:18.514216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:18.515794 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.516208 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.516209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.516210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.516275 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.516532 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.516533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.516534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.516656 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:18.516730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.517003 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.517088 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:42:18.517380 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.508213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:38555)
0809 16:42:18.508416 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.508456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.508983 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.509298 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.509301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.509898 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:18.510094 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.510098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.510776 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.510776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.511598 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.511600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.511955 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.511956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.512063 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.512587 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:18.512601 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:18.514096 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:18.515899 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.515906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.515908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.516132 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.516344 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:18.516345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:18.516345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:18.516458 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:18.516720 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:18.516727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:18.516867 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.516968 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.517128 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":116,"threads_started":1}
I20260809 16:42:18.517830 14972 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:18.517930 14972 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:18.518079 14972 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:18.518440 14423 ts_manager.cc:194] Registered new tserver with Master: 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883)
I20260809 16:42:18.518832 14423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:38555
I20260809 16:42:18.525938 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:42:18.533355 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.527344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.28.62.190:35167)
0809 16:42:18.527397 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.527398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.527522 (+ 124us) server_negotiation.cc:407] Connection header received
0809 16:42:18.527576 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.527577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.527585 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.527634 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:18.528213 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.528743 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.529480 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.529657 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.529709 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.531095 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:18.531096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:18.531097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:18.531105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:18.532392 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.532708 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.532709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.532709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.532763 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:18.532922 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:18.532923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:18.532923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:18.533063 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:18.533142 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.533228 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.533292 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 16:42:18.535303 14423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:42:18.536912 14423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:42:18.543586 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:32883 (local address 127.0.0.1:38640)
0809 16:42:18.539889 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.539900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.539952 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.540219 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.540220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.540222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:18.540279 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.540281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.541500 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.541501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.542412 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.542414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.543253 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.543345 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543359 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.543455 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.543512 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":80,"threads_started":1}
I20260809 16:42:18.543597 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.28.62.131:32883)
0809 16:42:18.539768 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.539773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.540011 (+ 238us) server_negotiation.cc:407] Connection header received
0809 16:42:18.540055 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.540058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.540075 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.540150 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:18.540367 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.541390 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.542506 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.542988 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543252 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543323 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.543472 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.543513 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
I20260809 16:42:18.543733 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.0.0.1:57554)
0809 16:42:18.539378 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.539418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.539482 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.539854 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.539857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.539859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:18.539933 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.539935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.541253 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.541254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.542442 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.542445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.543453 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.543530 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543536 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.543588 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.543638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":42,"thread_start_us":54,"threads_started":1}
I20260809 16:42:18.543964 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.28.62.129:42057)
0809 16:42:18.539580 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.539585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.539597 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:42:18.539642 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.539644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.539661 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.539753 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:18.540018 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.541153 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.542545 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.543291 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.543853 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.543892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260809 16:42:18.544306 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.28.62.130:36961)
0809 16:42:18.539826 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.539830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.539840 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:42:18.539953 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.539955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.539972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.540044 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:18.540254 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.540967 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.542544 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.543291 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543415 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.544208 (+ 793us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.544244 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":112,"threads_started":1}
I20260809 16:42:18.544353 14894 tablet_service.cc:1467] Processing CreateTablet for tablet 406a5e913e4449f0bd26d8254b29f187 (DEFAULT_TABLE table=test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:18.544538 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.539248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36961 (local address 127.0.0.1:47030)
0809 16:42:18.539758 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.539769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.539849 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.540125 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.540125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.540127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:18.540168 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.540169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.541069 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.541070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.542443 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.542445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.543455 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.543455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.543530 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.543536 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.543588 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.544465 (+ 877us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":47,"thread_start_us":73,"threads_started":1}
I20260809 16:42:18.544720 14894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406a5e913e4449f0bd26d8254b29f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:18.544632 14574 tablet_service.cc:1467] Processing CreateTablet for tablet 406a5e913e4449f0bd26d8254b29f187 (DEFAULT_TABLE table=test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:18.544952 14574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406a5e913e4449f0bd26d8254b29f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:18.545183 14737 tablet_service.cc:1467] Processing CreateTablet for tablet 406a5e913e4449f0bd26d8254b29f187 (DEFAULT_TABLE table=test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:18.545543 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406a5e913e4449f0bd26d8254b29f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:18.548430 14992 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap starting.
I20260809 16:42:18.548673 14990 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Bootstrap starting.
I20260809 16:42:18.548915 14994 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Bootstrap starting.
I20260809 16:42:18.549362 14990 tablet_bootstrap.cc:654] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:18.549574 14994 tablet_bootstrap.cc:654] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:18.549902 14990 log.cc:824] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:18.550101 14994 log.cc:824] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:18.550770 14990 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: No bootstrap required, opened a new log
I20260809 16:42:18.550799 14992 tablet_bootstrap.cc:654] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:18.550849 14990 ts_tablet_manager.cc:1397] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:42:18.550846 14994 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: No bootstrap required, opened a new log
I20260809 16:42:18.550917 14994 ts_tablet_manager.cc:1397] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:42:18.551375 14992 log.cc:824] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:18.552186 14992 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: No bootstrap required, opened a new log
I20260809 16:42:18.552256 14992 ts_tablet_manager.cc:1397] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260809 16:42:18.552649 14994 raft_consensus.cc:348] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.552855 14994 raft_consensus.cc:374] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:18.552901 14994 raft_consensus.cc:729] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b510862baef34f859fb6d2f6f463c9f2, State: Initialized, Role: FOLLOWER
I20260809 16:42:18.553013 14994 consensus_queue.cc:260] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.552935 14990 raft_consensus.cc:348] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.553141 14990 raft_consensus.cc:374] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:18.553182 14990 raft_consensus.cc:729] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73761d49041b463aa59438092f4903e9, State: Initialized, Role: FOLLOWER
I20260809 16:42:18.553285 14990 consensus_queue.cc:260] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.553318 14994 ts_tablet_manager.cc:1428] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:42:18.553450 14813 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
I20260809 16:42:18.553571 14990 ts_tablet_manager.cc:1428] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:42:18.553707 14972 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
I20260809 16:42:18.554251 14992 raft_consensus.cc:348] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.554446 14992 raft_consensus.cc:374] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:18.554500 14992 raft_consensus.cc:729] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1f7b3e65d0448b90e74a8674a0982d, State: Initialized, Role: FOLLOWER
I20260809 16:42:18.554627 14992 consensus_queue.cc:260] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.554929 14992 ts_tablet_manager.cc:1428] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 16:42:18.554952 14649 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
W20260809 16:42:18.593096 14814 tablet.cc:2369] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:42:18.687166 14650 tablet.cc:2369] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:42:18.758155 14973 tablet.cc:2369] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:42:18.772087 14998 raft_consensus.cc:482] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:18.772221 14998 raft_consensus.cc:504] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.772640 14998 leader_election.cc:288] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057), 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883)
I20260809 16:42:18.775671 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.772902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:38251 (local address 127.28.62.129:42057)
0809 16:42:18.772969 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.772971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.772982 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:42:18.773069 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.773071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.773084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.773144 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:18.773328 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.773885 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.774920 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.775296 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.775352 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.775359 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.775538 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.775584 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:42:18.775681 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.772811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.28.62.130:38251)
0809 16:42:18.772899 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.772931 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.773000 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.773155 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.773156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.773158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:18.773248 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.773251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.773971 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.773972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.774806 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.774809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.775372 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.775373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.775464 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.775468 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.775523 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.775560 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":34}
I20260809 16:42:18.775734 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.773068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:40161 (local address 127.28.62.131:32883)
0809 16:42:18.773116 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.773118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.773199 (+ 81us) server_negotiation.cc:407] Connection header received
0809 16:42:18.773237 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.773238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.773248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.773313 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:18.773477 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.773958 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.774883 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.775369 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.775442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.775451 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.775608 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.775651 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:42:18.775785 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.772932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:32883 (local address 127.28.62.130:40161)
0809 16:42:18.773135 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.773144 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:18.773194 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:18.773329 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:18.773330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:18.773332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:18.773408 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.773409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.774042 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.774043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.774799 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:18.774801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.775453 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.775454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:18.775524 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.775529 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:42:18.775587 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:42:18.775622 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260809 16:42:18.776089 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406a5e913e4449f0bd26d8254b29f187" candidate_uuid: "b510862baef34f859fb6d2f6f463c9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73761d49041b463aa59438092f4903e9" is_pre_election: true
I20260809 16:42:18.776090 14596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406a5e913e4449f0bd26d8254b29f187" candidate_uuid: "b510862baef34f859fb6d2f6f463c9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" is_pre_election: true
I20260809 16:42:18.776247 14917 raft_consensus.cc:2454] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b510862baef34f859fb6d2f6f463c9f2 in term 0.
I20260809 16:42:18.776247 14596 raft_consensus.cc:2454] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b510862baef34f859fb6d2f6f463c9f2 in term 0.
I20260809 16:42:18.776489 14676 leader_election.cc:302] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2; no voters:
I20260809 16:42:18.776664 14998 raft_consensus.cc:2781] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:42:18.776749 14998 raft_consensus.cc:482] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:42:18.776787 14998 raft_consensus.cc:3037] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:18.777544 14998 raft_consensus.cc:504] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.777698 14998 leader_election.cc:288] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057), 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883)
I20260809 16:42:18.777863 14596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406a5e913e4449f0bd26d8254b29f187" candidate_uuid: "b510862baef34f859fb6d2f6f463c9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
I20260809 16:42:18.777884 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406a5e913e4449f0bd26d8254b29f187" candidate_uuid: "b510862baef34f859fb6d2f6f463c9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73761d49041b463aa59438092f4903e9"
I20260809 16:42:18.777956 14917 raft_consensus.cc:3037] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:18.777952 14596 raft_consensus.cc:3037] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:18.778849 14917 raft_consensus.cc:2454] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b510862baef34f859fb6d2f6f463c9f2 in term 1.
I20260809 16:42:18.778849 14596 raft_consensus.cc:2454] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b510862baef34f859fb6d2f6f463c9f2 in term 1.
I20260809 16:42:18.779047 14676 leader_election.cc:302] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2; no voters:
I20260809 16:42:18.779150 14998 raft_consensus.cc:2781] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:42:18.779312 14998 raft_consensus.cc:686] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 1 LEADER]: Becoming Leader. State: Replica: b510862baef34f859fb6d2f6f463c9f2, State: Running, Role: LEADER
I20260809 16:42:18.779407 14998 consensus_queue.cc:237] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:18.780246 14423 catalog_manager.cc:5495] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130). New cstate: current_term: 1 leader_uuid: "b510862baef34f859fb6d2f6f463c9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20260809 16:42:18.829246 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.826646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.28.62.130:36961)
0809 16:42:18.826700 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:18.826702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:18.826710 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:42:18.826751 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:18.826752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:18.826770 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:18.826843 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:18.826952 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.827568 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.828492 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:18.828711 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:18.828781 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:18.828947 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:18.829060 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:42:18.829127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:18.829159 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 16:42:18.829183 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:42:18.831213 14917 raft_consensus.cc:1261] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Refusing update from remote peer b510862baef34f859fb6d2f6f463c9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:18.831220 14596 raft_consensus.cc:1261] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Refusing update from remote peer b510862baef34f859fb6d2f6f463c9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:18.831621 15002 consensus_queue.cc:1035] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:18.831776 14998 consensus_queue.cc:1035] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:18.834718 15008 mvcc.cc:205] Tried to move back new op lower bound from 7316659154249420800 to 7316659154041106432. Current Snapshot: MvccSnapshot[applied={T|T < 7316659154249420800}]
I20260809 16:42:18.835148 15007 mvcc.cc:205] Tried to move back new op lower bound from 7316659154249420800 to 7316659154041106432. Current Snapshot: MvccSnapshot[applied={T|T < 7316659154249420800}]
I20260809 16:42:18.836793 14423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:18.836886 14423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:18.837611 14423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Sending DeleteTablet for 3 replicas of tablet 406a5e913e4449f0bd26d8254b29f187
I20260809 16:42:18.837968 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 14500
I20260809 16:42:18.837966 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet 406a5e913e4449f0bd26d8254b29f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:18 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:47030
I20260809 16:42:18.838061 14574 tablet_service.cc:1514] Processing DeleteTablet for tablet 406a5e913e4449f0bd26d8254b29f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:18 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:57554
I20260809 16:42:18.838074 14894 tablet_service.cc:1514] Processing DeleteTablet for tablet 406a5e913e4449f0bd26d8254b29f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:18 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:38640
I20260809 16:42:18.841883 15016 tablet_replica.cc:330] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: stopping tablet replica
I20260809 16:42:18.841995 15016 raft_consensus.cc:2229] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:42:18.842092 15016 raft_consensus.cc:2258] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:18.842583 15016 ts_tablet_manager.cc:1905] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:18.844712 15016 ts_tablet_manager.cc:1918] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:18.844818 15016 log.cc:1196] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/406a5e913e4449f0bd26d8254b29f187
I20260809 16:42:18.845093 15016 ts_tablet_manager.cc:1939] T 406a5e913e4449f0bd26d8254b29f187 P b510862baef34f859fb6d2f6f463c9f2: Deleting consensus metadata
I20260809 16:42:18.845603 14405 catalog_manager.cc:4841] TS b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): tablet 406a5e913e4449f0bd26d8254b29f187 (table test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]) successfully deleted
I20260809 16:42:18.847162 15017 tablet_replica.cc:330] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: stopping tablet replica
I20260809 16:42:18.847303 15017 raft_consensus.cc:2229] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:18.847429 15017 raft_consensus.cc:2258] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:18.847846 15017 ts_tablet_manager.cc:1905] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:18.849373 15017 ts_tablet_manager.cc:1918] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:18.849474 15017 log.cc:1196] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/406a5e913e4449f0bd26d8254b29f187
I20260809 16:42:18.849792 15017 ts_tablet_manager.cc:1939] T 406a5e913e4449f0bd26d8254b29f187 P 73761d49041b463aa59438092f4903e9: Deleting consensus metadata
W20260809 16:42:18.849857 14404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:42057 (error 108)
I20260809 16:42:18.850347 14404 catalog_manager.cc:4841] TS 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): tablet 406a5e913e4449f0bd26d8254b29f187 (table test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]) successfully deleted
I20260809 16:42:18.850415 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:18.850284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.0.0.1:57568)
0809 16:42:18.850326 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:18.850364 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 16:42:18.850592 14404 catalog_manager.cc:4584] TS 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 406a5e913e4449f0bd26d8254b29f187: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42057: connect: Connection refused (error 111)
I20260809 16:42:18.851131 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 14654
I20260809 16:42:18.863351 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 14819
I20260809 16:42:18.870481 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 14383
I20260809 16:42:38.880040 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--webserver_interface=127.28.62.190
--webserver_port=39247
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:38.966488 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:38.966712 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:38.966737 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:38.967980 15019 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 16:42:38.968025 15019 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 16:42:38.968039 15019 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 16:42:38.968051 15019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:42:38.968091 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:38.968125 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:38.968142 15019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:42:38.968168 15019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:42:38.969681 15019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:38.969944 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:38.970216 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:38.980338 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:38.980448 15019 server_base.cc:1034] running on GCE node
W20260809 16:42:38.980338 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:38.980530 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:38.980897 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:38.981200 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:38.982368 15019 hybrid_clock.cc:648] HybridClock initialized: now 1786293758982345 us; error 34 us; skew 500 ppm
Aug 09 16:42:38 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293758, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:38.990183 15019 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:42:38.990707 15019 webserver.cc:460] Webserver started at http://127.28.62.190:39247/ using document root <none> and password file <none>
I20260809 16:42:38.990965 15019 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:38.991088 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:38.992616 15019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:38.993328 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:38.993513 15019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:42:38.993628 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "57be2e0243764193b055b5ad5a7cc7fe"
format_stamp: "Formatted at 2026-08-09 16:42:17 on dist-test-slave-wl2h"
server_key: "abbcf768a852e43bd72955ec98a67972"
server_key_iv: "40a3da58cc92965503b515b1b7335b80"
server_key_version: "encryptionkey@0"
I20260809 16:42:38.994122 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:39.007427 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:39.009549 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:39.009744 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:39.014380 15019 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35167
I20260809 16:42:39.014415 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35167 every 8 connection(s)
I20260809 16:42:39.014838 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:42:39.016505 15105 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:42:39.016985 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15019
I20260809 16:42:39.017207 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap starting.
I20260809 16:42:39.017266 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:42057
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=40413
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:39.028436 15105 log.cc:824] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:39.031414 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:39.031819 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap complete.
I20260809 16:42:39.035518 15105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:39.035841 15105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Initialized, Role: FOLLOWER
I20260809 16:42:39.036000 15105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:39.036063 15105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:42:39.036111 15105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:42:39.036159 15105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 1 FOLLOWER]: Advancing to term 2
I20260809 16:42:39.037483 15105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:39.037591 15105 leader_election.cc:302] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57be2e0243764193b055b5ad5a7cc7fe; no voters:
I20260809 16:42:39.037763 15105 leader_election.cc:288] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 16:42:39.037850 15109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Leader election won for term 2
I20260809 16:42:39.038012 15105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:42:39.038082 15109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 LEADER]: Becoming Leader. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Running, Role: LEADER
I20260809 16:42:39.038216 15109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:39.038554 15110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:39.038626 15110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:39.038869 15112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57be2e0243764193b055b5ad5a7cc7fe. Latest consensus state: current_term: 2 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:39.039039 15112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:39.039112 15120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:42:39.039911 15120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:42:39.040012 15120 catalog_manager.cc:1259] Loaded cluster ID: 2227970d8c25468db5777fad11521bad
I20260809 16:42:39.040046 15120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:42:39.041052 15120 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:42:39.041255 15120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Loaded TSK: 0
I20260809 16:42:39.049057 15120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Generated new TSK 1
I20260809 16:42:39.049165 15120 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 16:42:39.122520 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:39.122726 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:39.122749 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:39.123936 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:39.123980 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:39.124006 15107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:42:39.125443 15107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:42057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=40413
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:39.125715 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:39.125979 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:39.132434 15139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.132450 15141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.132683 15138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:39.132514 15107 server_base.cc:1034] running on GCE node
I20260809 16:42:39.132901 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:39.133168 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:39.134343 15107 hybrid_clock.cc:648] HybridClock initialized: now 1786293759134329 us; error 32 us; skew 500 ppm
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:39.136932 15107 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:42:39.137454 15107 webserver.cc:460] Webserver started at http://127.28.62.129:40413/ using document root <none> and password file <none>
I20260809 16:42:39.137712 15107 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:39.137770 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:39.139406 15107 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:39.140146 15150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.140352 15107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:39.140460 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "50aae5b2b806a3c3d49ae5a227866ae2"
server_key_iv: "24781f8a9cee383fc0d61ab596d411b0"
server_key_version: "encryptionkey@0"
I20260809 16:42:39.140944 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:39.156351 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:39.158939 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:39.159129 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:39.159400 15107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:39.159932 15157 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 16:42:39.161442 15107 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 16:42:39.161505 15107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 16:42:39.161541 15107 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 16:42:39.162479 15107 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 16:42:39.162529 15107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:39.162642 15157 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap starting.
I20260809 16:42:39.169997 15107 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:42057
I20260809 16:42:39.170275 15287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:42057 every 8 connection(s)
I20260809 16:42:39.170460 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:42:39.173925 15157 log.cc:824] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:39.176682 15157 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:39.176985 15157 tablet_bootstrap.cc:492] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap complete.
I20260809 16:42:39.177688 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15107
I20260809 16:42:39.177978 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:36961
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=33651
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:39.178356 15157 ts_tablet_manager.cc:1397] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260809 16:42:39.180265 15157 raft_consensus.cc:348] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:39.180645 15157 raft_consensus.cc:729] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1f7b3e65d0448b90e74a8674a0982d, State: Initialized, Role: FOLLOWER
I20260809 16:42:39.180787 15157 consensus_queue.cc:260] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } } peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } }
I20260809 16:42:39.181094 15157 ts_tablet_manager.cc:1428] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:39.186548 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.171261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:49527)
0809 16:42:39.171460 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.171528 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.172305 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.174730 (+ 2425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.174734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.175139 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:39.175410 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.175418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.177165 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.177166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.178576 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.178579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.179468 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.179470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.179602 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.180282 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:39.180292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:39.181951 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:39.184572 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.184580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.184583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.184891 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.185226 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.185227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.185228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.185367 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.185874 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:39.185881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:39.186035 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.186157 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.186328 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260809 16:42:39.186825 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.171553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:49527 (local address 127.28.62.190:35167)
0809 16:42:39.174271 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.174281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.174324 (+ 43us) server_negotiation.cc:407] Connection header received
0809 16:42:39.174435 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.174440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.174510 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.174677 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.175564 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.177012 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.178713 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.179336 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.179935 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.182078 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.182083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.182083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.182097 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.184456 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.184988 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.184992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.184993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.185127 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.185448 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.185449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.185450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.185789 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.185889 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.186242 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.186404 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2645,"thread_start_us":87,"threads_started":1}
I20260809 16:42:39.187770 15288 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:39.187883 15288 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:39.188088 15288 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:39.188957 15064 ts_manager.cc:194] Registered new tserver with Master: 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057)
I20260809 16:42:39.190295 15064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:49527
I20260809 16:42:39.191423 15288 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293758, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.129@KRBTEST.COM
I20260809 16:42:39.200644 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.189951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.28.62.129:42057)
0809 16:42:39.190247 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.190251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.190260 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:42:39.190294 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.190296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.190309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.191067 (+ 758us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.191998 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.193175 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.194338 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.194793 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.194864 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.197354 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.197363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.197363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.197372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.199286 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.199731 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.199734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.199735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.199802 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.200019 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.200020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.200020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.200195 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.200279 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.200471 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.200556 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":75,"threads_started":1}
I20260809 16:42:39.200649 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.189691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.0.0.1:33914)
0809 16:42:39.189980 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.190023 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.190094 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.190517 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.190524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.191852 (+ 1328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:39.191910 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.191913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.193267 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.193268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.194270 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.194273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.194874 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.194875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.194952 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.195528 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:39.195533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:39.197256 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:39.199372 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.199379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.199381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.199637 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.199877 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.199878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.199878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.199947 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.200269 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:39.200276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:39.200329 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.200415 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.200536 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260809 16:42:39.201397 15212 tablet_service.cc:1514] Processing DeleteTablet for tablet 406a5e913e4449f0bd26d8254b29f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:18 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33914
I20260809 16:42:39.202021 15298 tablet_replica.cc:330] stopping tablet replica
I20260809 16:42:39.202214 15298 raft_consensus.cc:2229] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:39.202307 15298 raft_consensus.cc:2258] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:39.202649 15298 ts_tablet_manager.cc:1905] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:39.204995 15298 ts_tablet_manager.cc:1918] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:39.205168 15298 log.cc:1196] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal/wals/406a5e913e4449f0bd26d8254b29f187
I20260809 16:42:39.205439 15298 ts_tablet_manager.cc:1939] T 406a5e913e4449f0bd26d8254b29f187 P 8c1f7b3e65d0448b90e74a8674a0982d: Deleting consensus metadata
I20260809 16:42:39.206013 15043 catalog_manager.cc:4841] TS 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057): tablet 406a5e913e4449f0bd26d8254b29f187 (table test-table [id=bfe7ae1ee5214bd6a2602a51a866c79b]) successfully deleted
W20260809 16:42:39.269188 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:39.269397 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:39.269419 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:39.270591 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:39.270638 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:39.270673 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:42:39.272192 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:36961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=33651
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:39.272460 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:39.272701 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:39.277853 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.277880 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.277860 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:39.278038 15294 server_base.cc:1034] running on GCE node
I20260809 16:42:39.278280 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:39.278499 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:39.279704 15294 hybrid_clock.cc:648] HybridClock initialized: now 1786293759279684 us; error 31 us; skew 500 ppm
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:39.282591 15294 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:42:39.283145 15294 webserver.cc:460] Webserver started at http://127.28.62.130:33651/ using document root <none> and password file <none>
I20260809 16:42:39.283417 15294 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:39.283504 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:39.285175 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:42:39.285856 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.286052 15294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:39.286156 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b510862baef34f859fb6d2f6f463c9f2"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "7be4ca700d1c48d2b7a22c47feebc8cd"
server_key_iv: "970764e26ab99731c0f9393f62769ee5"
server_key_version: "encryptionkey@0"
I20260809 16:42:39.286590 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:39.303073 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:39.305087 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:39.305262 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:39.305508 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:39.305840 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:39.305902 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.305936 15294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:39.305968 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.312192 15294 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36961
I20260809 16:42:39.312306 15456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36961 every 8 connection(s)
I20260809 16:42:39.312790 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:42:39.316282 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15294
I20260809 16:42:39.316618 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:32883
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=35683
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:39.324610 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.313786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:47837 (local address 127.28.62.190:35167)
0809 16:42:39.313855 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.313857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.314090 (+ 233us) server_negotiation.cc:407] Connection header received
0809 16:42:39.314659 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.314661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.314677 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.314722 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.315812 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.316351 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.317478 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.317697 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.317786 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.321153 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.321155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.321155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.321163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.322701 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.323438 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.323440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.323440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.323508 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.323859 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.323860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.323861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.323989 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.324041 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.324452 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.324538 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 16:42:39.324939 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.313765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:47837)
0809 16:42:39.313961 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.313999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.314536 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.314799 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.314802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.315483 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:39.315704 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.315709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.316466 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.316466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.317384 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.317387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.317801 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.317801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.317935 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.318673 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:39.318692 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:39.320980 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:39.322812 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.322823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.322826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.323281 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.323593 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.323594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.323594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.323769 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.324067 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:39.324076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:39.324282 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.324412 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.324624 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260809 16:42:39.325541 15458 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:39.325664 15458 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:39.325902 15458 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:39.326519 15064 ts_manager.cc:194] Registered new tserver with Master: b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
I20260809 16:42:39.326921 15064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:47837
W20260809 16:42:39.410332 15461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:39.410552 15461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:39.410576 15461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:39.411830 15461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:39.411876 15461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:39.411904 15461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:42:39.413384 15461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:32883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=35683
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:39.413673 15461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:39.413918 15461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:39.419252 15472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.419260 15468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:39.419384 15467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:39.420437 15461 server_base.cc:1034] running on GCE node
I20260809 16:42:39.420717 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:39.420984 15461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:39.422214 15461 hybrid_clock.cc:648] HybridClock initialized: now 1786293759422186 us; error 34 us; skew 500 ppm
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:39.426080 15461 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:42:39.426956 15461 webserver.cc:460] Webserver started at http://127.28.62.131:35683/ using document root <none> and password file <none>
I20260809 16:42:39.427275 15461 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:39.427345 15461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:39.428779 15461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:42:39.429507 15482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.429725 15461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:42:39.429832 15461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "73761d49041b463aa59438092f4903e9"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "e9e401744360f6eafbeec267875f6567"
server_key_iv: "1622c63315a8eac84e2ae60fec8f6e84"
server_key_version: "encryptionkey@0"
I20260809 16:42:39.430193 15461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:39.444319 15461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:39.446655 15461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:39.446781 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:39.447000 15461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:39.447350 15461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:42:39.447382 15461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.447405 15461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:42:39.447417 15461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:39.453305 15461 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:32883
I20260809 16:42:39.453488 15616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:32883 every 8 connection(s)
I20260809 16:42:39.453720 15461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:42:39.456914 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15461
Aug 09 16:42:39 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293759, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:42:39.464521 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.454533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:58753 (local address 127.28.62.190:35167)
0809 16:42:39.454596 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.454598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.454840 (+ 242us) server_negotiation.cc:407] Connection header received
0809 16:42:39.455400 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.455401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.455413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.455486 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.456461 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.456984 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.458236 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.458442 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.458530 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.461161 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.461163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.461163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.461171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.462853 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.463400 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.463400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.463401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.463454 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.463765 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.463765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.463766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.463943 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.464019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.464374 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.464455 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 16:42:39.464766 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.454491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:58753)
0809 16:42:39.454665 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.454708 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.455296 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.455561 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.455565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.456166 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:42:39.456361 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.456365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.457127 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.457128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.458124 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.458127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.458542 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.458543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.458675 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.459279 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:42:39.459289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:42:39.461060 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:42:39.462954 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.462962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.462964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.463320 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.463527 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:42:39.463528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:42:39.463528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:42:39.463678 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:42:39.464016 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:42:39.464020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:42:39.464206 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.464310 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.464509 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260809 16:42:39.465271 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.457396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.28.62.190:35167)
0809 16:42:39.457535 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.457537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.457604 (+ 67us) server_negotiation.cc:407] Connection header received
0809 16:42:39.457683 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.457684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.457696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.457777 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.458388 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.458901 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.460125 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.460439 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.460521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.461882 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.461884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.461884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.461891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.463642 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.464087 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.464088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.464089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.464150 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.464365 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.464366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.464366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.464504 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.464677 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.465043 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.465142 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260809 16:42:39.465351 15617 heartbeater.cc:347] Connected to a master server at 127.28.62.190:35167
I20260809 16:42:39.465461 15617 heartbeater.cc:464] Registering TS with master...
I20260809 16:42:39.465649 15617 heartbeater.cc:511] Master 127.28.62.190:35167 requested a full tablet report, sending...
I20260809 16:42:39.466111 15064 ts_manager.cc:194] Registered new tserver with Master: 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883)
I20260809 16:42:39.466444 15064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:58753
I20260809 16:42:39.477082 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:42:39.480296 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.477487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.28.62.190:35167)
0809 16:42:39.477577 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.477579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.477694 (+ 115us) server_negotiation.cc:407] Connection header received
0809 16:42:39.477789 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.477790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.477836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.477915 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:39.478070 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.478565 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.479459 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.479678 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.479759 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.479933 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:39.479988 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 16:42:39.480190 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49056: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 16:42:39.480381 15623 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49056: token expired
W20260809 16:42:39.480522 14976 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.62.190:35167)
I20260809 16:42:39.487773 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.480820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.28.62.190:35167)
0809 16:42:39.480884 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.480887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.480896 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:42:39.480960 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.480961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.480970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.481042 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:42:39.481546 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.481998 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.482747 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.482931 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.483005 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.484415 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:42:39.484417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:42:39.484417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:42:39.484424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:42:39.486449 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.486876 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.486877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.486878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.486954 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:42:39.487191 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:42:39.487192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:42:39.487193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:42:39.487313 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:42:39.487442 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.487572 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.487665 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:42:39.489190 15064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:42:39.491132 15064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:42:39.493748 15212 tablet_service.cc:1467] Processing CreateTablet for tablet f250c3e96a284de5ac04de967b61f7f3 (DEFAULT_TABLE table=test-table [id=11df02c6bbe3481c9e40490cfbc7bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:39.493980 15212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250c3e96a284de5ac04de967b61f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:39.495573 15157 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap starting.
I20260809 16:42:39.495929 15157 tablet_bootstrap.cc:654] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:39.496804 15157 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: No bootstrap required, opened a new log
I20260809 16:42:39.496879 15157 ts_tablet_manager.cc:1397] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:39.497414 15157 raft_consensus.cc:348] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.497491 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.493088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:32883 (local address 127.0.0.1:38140)
0809 16:42:39.493596 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.493606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.493657 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.494085 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.494086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.494088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:39.494127 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.494129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.494977 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.494978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.496295 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.496298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.497218 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.497219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.497284 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.497306 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.497365 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.497413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":251,"thread_start_us":81,"threads_started":1}
I20260809 16:42:39.497502 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.493349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.28.62.131:32883)
0809 16:42:39.493575 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.493580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.493664 (+ 84us) server_negotiation.cc:407] Connection header received
0809 16:42:39.493890 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.493893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.493911 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.493983 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:39.494188 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.494891 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.496384 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.497110 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.497215 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.497279 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.497381 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.497422 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260809 16:42:39.497515 15157 raft_consensus.cc:374] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:39.497578 15157 raft_consensus.cc:729] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1f7b3e65d0448b90e74a8674a0982d, State: Initialized, Role: FOLLOWER
I20260809 16:42:39.497635 15157 consensus_queue.cc:260] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.497743 15157 ts_tablet_manager.cc:1428] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:39.498235 15540 tablet_service.cc:1467] Processing CreateTablet for tablet f250c3e96a284de5ac04de967b61f7f3 (DEFAULT_TABLE table=test-table [id=11df02c6bbe3481c9e40490cfbc7bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:39.498617 15540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250c3e96a284de5ac04de967b61f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:39.498852 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.493088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36961 (local address 127.0.0.1:56124)
0809 16:42:39.493128 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.493139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.493249 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.493736 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.493737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.493738 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:39.493791 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.493793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.495068 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.495069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.496320 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.496324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.498610 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.498610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.498694 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.498698 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.498753 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.498804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":66}
I20260809 16:42:39.498907 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.493329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.28.62.130:36961)
0809 16:42:39.493530 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.493535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.493569 (+ 34us) server_negotiation.cc:407] Connection header received
0809 16:42:39.493618 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.493620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.493637 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.493731 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:39.493848 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.494917 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.496415 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.498517 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.498610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.498665 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.498777 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.498826 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":119,"threads_started":1}
I20260809 16:42:39.499526 15376 tablet_service.cc:1467] Processing CreateTablet for tablet f250c3e96a284de5ac04de967b61f7f3 (DEFAULT_TABLE table=test-table [id=11df02c6bbe3481c9e40490cfbc7bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:42:39.499825 15376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f250c3e96a284de5ac04de967b61f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:42:39.501591 15629 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Bootstrap starting.
I20260809 16:42:39.502264 15629 tablet_bootstrap.cc:654] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:39.502506 15630 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Bootstrap starting.
I20260809 16:42:39.502889 15629 log.cc:824] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:39.503149 15630 tablet_bootstrap.cc:654] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Neither blocks nor log segments found. Creating new log.
I20260809 16:42:39.503643 15629 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: No bootstrap required, opened a new log
I20260809 16:42:39.503690 15630 log.cc:824] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:39.503741 15629 ts_tablet_manager.cc:1397] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:42:39.504498 15630 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: No bootstrap required, opened a new log
I20260809 16:42:39.504549 15630 ts_tablet_manager.cc:1397] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:42:39.505460 15629 raft_consensus.cc:348] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.505622 15629 raft_consensus.cc:374] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:39.505677 15629 raft_consensus.cc:729] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73761d49041b463aa59438092f4903e9, State: Initialized, Role: FOLLOWER
I20260809 16:42:39.505801 15629 consensus_queue.cc:260] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.506067 15629 ts_tablet_manager.cc:1428] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 16:42:39.506131 15617 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
I20260809 16:42:39.506065 15630 raft_consensus.cc:348] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.506242 15630 raft_consensus.cc:374] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:42:39.506285 15630 raft_consensus.cc:729] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b510862baef34f859fb6d2f6f463c9f2, State: Initialized, Role: FOLLOWER
I20260809 16:42:39.506397 15630 consensus_queue.cc:260] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.506705 15458 heartbeater.cc:503] Master 127.28.62.190:35167 was elected leader, sending a full tablet report...
I20260809 16:42:39.506686 15630 ts_tablet_manager.cc:1428] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:42:39.531260 15633 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:42:39.531391 15633 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.531827 15633 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961), 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057)
I20260809 16:42:39.535144 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.532026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36961 (local address 127.28.62.131:55017)
0809 16:42:39.532125 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.532139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.532203 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.532373 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.532374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.532376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:39.532458 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.532460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.533249 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.533250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.534097 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.534099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.534848 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.534849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.534935 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.534939 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.534999 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.535061 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 16:42:39.535148 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.532190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:55017 (local address 127.28.62.130:36961)
0809 16:42:39.532244 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.532247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.532255 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:42:39.532283 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.532284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.532293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.532370 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:39.532531 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.533145 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.534237 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.534765 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.534821 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.534828 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.535048 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.535086 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:42:39.535718 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.532107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:59563 (local address 127.28.62.129:42057)
0809 16:42:39.532158 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.532160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.532407 (+ 247us) server_negotiation.cc:407] Connection header received
0809 16:42:39.532478 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.532479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.532509 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.532576 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:42:39.532764 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.533481 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.534618 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.535258 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.535329 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.535402 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.535634 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.535670 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:42:39.535740 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.532026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.28.62.131:59563)
0809 16:42:39.532329 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.532338 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:39.532416 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:39.532593 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:42:39.532594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:42:39.532595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:42:39.532679 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.532680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.533555 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.533556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.534498 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:42:39.534501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.535470 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.535471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:42:39.535527 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.535530 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:42:39.535618 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 16:42:39.535657 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":110,"thread_start_us":135,"threads_started":1}
I20260809 16:42:39.535712 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250c3e96a284de5ac04de967b61f7f3" candidate_uuid: "73761d49041b463aa59438092f4903e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b510862baef34f859fb6d2f6f463c9f2" is_pre_election: true
I20260809 16:42:39.535876 15402 raft_consensus.cc:2454] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73761d49041b463aa59438092f4903e9 in term 0.
I20260809 16:42:39.536003 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250c3e96a284de5ac04de967b61f7f3" candidate_uuid: "73761d49041b463aa59438092f4903e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" is_pre_election: true
I20260809 16:42:39.536119 15490 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2; no voters:
I20260809 16:42:39.536163 15234 raft_consensus.cc:2454] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73761d49041b463aa59438092f4903e9 in term 0.
I20260809 16:42:39.536245 15633 raft_consensus.cc:2781] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:42:39.536304 15633 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:42:39.536334 15633 raft_consensus.cc:3037] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:39.537077 15633 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.537233 15633 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [CANDIDATE]: Term 1 election: Requested vote from peers b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961), 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057)
I20260809 16:42:39.537449 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250c3e96a284de5ac04de967b61f7f3" candidate_uuid: "73761d49041b463aa59438092f4903e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b510862baef34f859fb6d2f6f463c9f2"
I20260809 16:42:39.537449 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f250c3e96a284de5ac04de967b61f7f3" candidate_uuid: "73761d49041b463aa59438092f4903e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
I20260809 16:42:39.537539 15234 raft_consensus.cc:3037] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:39.537539 15402 raft_consensus.cc:3037] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:42:39.538561 15234 raft_consensus.cc:2454] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73761d49041b463aa59438092f4903e9 in term 1.
I20260809 16:42:39.538561 15402 raft_consensus.cc:2454] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73761d49041b463aa59438092f4903e9 in term 1.
I20260809 16:42:39.538751 15488 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73761d49041b463aa59438092f4903e9, 8c1f7b3e65d0448b90e74a8674a0982d; no voters:
I20260809 16:42:39.538879 15633 raft_consensus.cc:2781] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:42:39.539062 15633 raft_consensus.cc:686] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 1 LEADER]: Becoming Leader. State: Replica: 73761d49041b463aa59438092f4903e9, State: Running, Role: LEADER
I20260809 16:42:39.539237 15633 consensus_queue.cc:237] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:39.540134 15064 catalog_manager.cc:5495] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73761d49041b463aa59438092f4903e9 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "73761d49041b463aa59438092f4903e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } health_report { overall_health: UNKNOWN } } }
I20260809 16:42:39.548743 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.546324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.28.62.131:32883)
0809 16:42:39.546394 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:42:39.546397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:42:39.546406 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:42:39.546443 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:42:39.546444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:42:39.546459 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:42:39.546531 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:42:39.546683 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.547236 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.547961 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:42:39.548154 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:42:39.548228 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:42:39.548394 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:42:39.548485 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:42:39.548573 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 16:42:39.548614 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 16:42:39.548634 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 16:42:39.550663 15234 raft_consensus.cc:1261] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Refusing update from remote peer 73761d49041b463aa59438092f4903e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:39.550699 15402 raft_consensus.cc:1261] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Refusing update from remote peer 73761d49041b463aa59438092f4903e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:42:39.551035 15633 consensus_queue.cc:1035] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:39.551237 15636 consensus_queue.cc:1035] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:42:39.553339 15639 mvcc.cc:205] Tried to move back new op lower bound from 7316659239116689408 to 7316659239073566720. Current Snapshot: MvccSnapshot[applied={T|T < 7316659239116689408}]
I20260809 16:42:39.553952 15640 mvcc.cc:205] Tried to move back new op lower bound from 7316659239116689408 to 7316659239073566720. Current Snapshot: MvccSnapshot[applied={T|T < 7316659239116689408}]
I20260809 16:42:39.554111 15641 mvcc.cc:205] Tried to move back new op lower bound from 7316659239116689408 to 7316659239073566720. Current Snapshot: MvccSnapshot[applied={T|T < 7316659239116689408 or (T in {7316659239116689408})}]
I20260809 16:42:39.556000 15063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:39.556106 15063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:42:39.556947 15063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Sending DeleteTablet for 3 replicas of tablet f250c3e96a284de5ac04de967b61f7f3
I20260809 16:42:39.557327 15212 tablet_service.cc:1514] Processing DeleteTablet for tablet f250c3e96a284de5ac04de967b61f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:39 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:33914
I20260809 16:42:39.557324 15376 tablet_service.cc:1514] Processing DeleteTablet for tablet f250c3e96a284de5ac04de967b61f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:39 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:56124
I20260809 16:42:39.557353 15540 tablet_service.cc:1514] Processing DeleteTablet for tablet f250c3e96a284de5ac04de967b61f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:42:39 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:38140
I20260809 16:42:39.557427 15298 tablet_replica.cc:330] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: stopping tablet replica
I20260809 16:42:39.557478 15298 raft_consensus.cc:2229] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:39.557533 15298 raft_consensus.cc:2258] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:39.557564 15652 tablet_replica.cc:330] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: stopping tablet replica
I20260809 16:42:39.557720 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15107
I20260809 16:42:39.557790 15652 raft_consensus.cc:2229] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:42:39.557822 15298 ts_tablet_manager.cc:1905] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:39.557909 15652 raft_consensus.cc:2258] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:39.558365 15652 ts_tablet_manager.cc:1905] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:39.560285 15652 ts_tablet_manager.cc:1918] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:39.560352 15652 log.cc:1196] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/f250c3e96a284de5ac04de967b61f7f3
I20260809 16:42:39.560645 15652 ts_tablet_manager.cc:1939] T f250c3e96a284de5ac04de967b61f7f3 P 73761d49041b463aa59438092f4903e9: Deleting consensus metadata
I20260809 16:42:39.560916 15651 tablet_replica.cc:330] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: stopping tablet replica
I20260809 16:42:39.561057 15651 raft_consensus.cc:2229] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:42:39.561196 15651 raft_consensus.cc:2258] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:42:39.561257 15043 catalog_manager.cc:4841] TS 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): tablet f250c3e96a284de5ac04de967b61f7f3 (table test-table [id=11df02c6bbe3481c9e40490cfbc7bb9d]) successfully deleted
I20260809 16:42:39.561724 15651 ts_tablet_manager.cc:1905] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:42:39.563838 15651 ts_tablet_manager.cc:1918] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:42:39.563974 15651 log.cc:1196] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/f250c3e96a284de5ac04de967b61f7f3
I20260809 16:42:39.564323 15651 ts_tablet_manager.cc:1939] T f250c3e96a284de5ac04de967b61f7f3 P b510862baef34f859fb6d2f6f463c9f2: Deleting consensus metadata
I20260809 16:42:39.564787 15046 catalog_manager.cc:4841] TS b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): tablet f250c3e96a284de5ac04de967b61f7f3 (table test-table [id=11df02c6bbe3481c9e40490cfbc7bb9d]) successfully deleted
W20260809 16:42:39.567417 15043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:42057 (error 108)
I20260809 16:42:39.567830 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15294
I20260809 16:42:39.567981 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:39.567825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:42057 (local address 127.0.0.1:33926)
0809 16:42:39.567900 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:39.567943 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 16:42:39.571789 15043 catalog_manager.cc:4584] TS 8c1f7b3e65d0448b90e74a8674a0982d (127.28.62.129:42057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f250c3e96a284de5ac04de967b61f7f3: Network error: Client connection negotiation failed: client connection to 127.28.62.129:42057: connect: Connection refused (error 111)
I20260809 16:42:39.584652 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15461
I20260809 16:42:39.591353 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15019
I20260809 16:42:59.599567 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--webserver_interface=127.28.62.190
--webserver_port=39247
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:59.683097 15655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:59.683312 15655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:59.683336 15655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:59.684413 15655 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 16:42:59.684445 15655 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 16:42:59.684458 15655 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 16:42:59.684470 15655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:42:59.684482 15655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:59.684495 15655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:59.684507 15655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:42:59.684520 15655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:42:59.685911 15655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:59.686102 15655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:59.686285 15655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:59.691275 15663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:59.691390 15655 server_base.cc:1034] running on GCE node
W20260809 16:42:59.691243 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:59.691259 15661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:59.691730 15655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:59.692039 15655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:59.693222 15655 hybrid_clock.cc:648] HybridClock initialized: now 1786293779693219 us; error 37 us; skew 500 ppm
Aug 09 16:42:59 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293779, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:59.699455 15655 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:42:59.699962 15655 webserver.cc:460] Webserver started at http://127.28.62.190:39247/ using document root <none> and password file <none>
I20260809 16:42:59.700248 15655 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:59.700325 15655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:59.701750 15655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:59.702442 15673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:59.702677 15655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:59.702813 15655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "57be2e0243764193b055b5ad5a7cc7fe"
format_stamp: "Formatted at 2026-08-09 16:42:17 on dist-test-slave-wl2h"
server_key: "abbcf768a852e43bd72955ec98a67972"
server_key_iv: "40a3da58cc92965503b515b1b7335b80"
server_key_version: "encryptionkey@0"
I20260809 16:42:59.703229 15655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:59.713862 15655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:59.715974 15655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:59.716144 15655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:59.720345 15655 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35167
I20260809 16:42:59.720407 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35167 every 8 connection(s)
I20260809 16:42:59.720804 15655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:42:59.722311 15733 sys_catalog.cc:263] Verifying existing consensus state
I20260809 16:42:59.722900 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap starting.
I20260809 16:42:59.726169 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15655
I20260809 16:42:59.726449 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:42057
--local_ip_for_outbound_sockets=127.28.62.129
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=40413
--webserver_interface=127.28.62.129
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:42:59.733230 15733 log.cc:824] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:59.735605 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:59.735909 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Bootstrap complete.
I20260809 16:42:59.738257 15733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:59.738507 15733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Initialized, Role: FOLLOWER
I20260809 16:42:59.738703 15733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:59.738785 15733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:42:59.738839 15733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:42:59.738890 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 2 FOLLOWER]: Advancing to term 3
I20260809 16:42:59.739941 15733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:59.740063 15733 leader_election.cc:302] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57be2e0243764193b055b5ad5a7cc7fe; no voters:
I20260809 16:42:59.740278 15733 leader_election.cc:288] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 16:42:59.740314 15738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 3 FOLLOWER]: Leader election won for term 3
I20260809 16:42:59.740433 15738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [term 3 LEADER]: Becoming Leader. State: Replica: 57be2e0243764193b055b5ad5a7cc7fe, State: Running, Role: LEADER
I20260809 16:42:59.740518 15738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } }
I20260809 16:42:59.740556 15733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:42:59.740792 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57be2e0243764193b055b5ad5a7cc7fe. Latest consensus state: current_term: 3 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:59.740859 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:59.740813 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57be2e0243764193b055b5ad5a7cc7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be2e0243764193b055b5ad5a7cc7fe" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35167 } } }
I20260809 16:42:59.741031 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe [sys.catalog]: This master's current role is: LEADER
I20260809 16:42:59.741058 15744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:42:59.741928 15744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:42:59.742051 15744 catalog_manager.cc:1259] Loaded cluster ID: 2227970d8c25468db5777fad11521bad
I20260809 16:42:59.742086 15744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:42:59.743161 15744 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:42:59.743372 15744 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Loaded TSK: 0
I20260809 16:42:59.743433 15744 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57be2e0243764193b055b5ad5a7cc7fe: Loaded TSK: 1
F20260809 16:42:59.743685 15744 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2f665969ad google::LogMessage::Fail() at ??:0
@ 0x7f2f6659988a google::LogMessage::SendToLog() at ??:0
@ 0x7f2f665964de google::LogMessage::Flush() at ??:0
@ 0x7f2f66597a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2f68d03104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2f68d00d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2f6e4f01cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2f6e4f048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2f6e4f0594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2f6e4b6c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2f6e4c0ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2f684640d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2f68459954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2f692bd6db start_thread at ??:0
@ 0x7f2f6509461f clone at ??:0
W20260809 16:42:59.831595 15735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:59.831800 15735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:59.831825 15735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:59.832985 15735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:59.833029 15735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:59.833057 15735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:42:59.834517 15735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:42057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=40413
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:59.834749 15735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:59.834997 15735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:59.840126 15785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:59.840126 15784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:59.840341 15735 server_base.cc:1034] running on GCE node
W20260809 16:42:59.840582 15788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:59.840806 15735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:59.841079 15735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:59.842264 15735 hybrid_clock.cc:648] HybridClock initialized: now 1786293779842240 us; error 44 us; skew 500 ppm
Aug 09 16:42:59 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293779, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:59.844944 15735 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:42:59.845475 15735 webserver.cc:460] Webserver started at http://127.28.62.129:40413/ using document root <none> and password file <none>
I20260809 16:42:59.845706 15735 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:59.845779 15735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:59.847231 15735 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:59.848047 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:59.848217 15735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:42:59.848321 15735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8c1f7b3e65d0448b90e74a8674a0982d"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "50aae5b2b806a3c3d49ae5a227866ae2"
server_key_iv: "24781f8a9cee383fc0d61ab596d411b0"
server_key_version: "encryptionkey@0"
I20260809 16:42:59.848678 15735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:42:59.855669 15735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:42:59.857722 15735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:42:59.857878 15735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:42:59.858129 15735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:42:59.858677 15804 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 16:42:59.860204 15735 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 16:42:59.860255 15735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260809 16:42:59.860291 15735 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 16:42:59.860951 15735 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 16:42:59.860996 15735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260809 16:42:59.861042 15804 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap starting.
I20260809 16:42:59.863065 15804 log.cc:824] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:42:59.866485 15804 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 16:42:59.866820 15804 tablet_bootstrap.cc:492] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Bootstrap complete.
I20260809 16:42:59.866998 15804 ts_tablet_manager.cc:1397] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260809 16:42:59.868678 15735 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:42057
I20260809 16:42:59.868789 15928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:42057 every 8 connection(s)
I20260809 16:42:59.869055 15735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:42:59.869285 15804 raft_consensus.cc:348] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:59.869659 15804 raft_consensus.cc:729] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1f7b3e65d0448b90e74a8674a0982d, State: Initialized, Role: FOLLOWER
I20260809 16:42:59.869828 15804 consensus_queue.cc:260] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:42:59.870090 15804 ts_tablet_manager.cc:1428] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 16:42:59.872570 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15735
I20260809 16:42:59.872881 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:36961
--local_ip_for_outbound_sockets=127.28.62.130
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=33651
--webserver_interface=127.28.62.130
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:42:59.947892 15931 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:42:59.869715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:45561)
0809 16:42:59.869893 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:59.869935 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:42:59.870088 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:42:59.947660 (+ 77572us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260809 16:42:59.948210 15929 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35167 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:35167: Client connection negotiation failed: client connection to 127.28.62.190:35167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 16:42:59.948782 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:59.948622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:55709)
0809 16:42:59.948700 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:59.948758 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 16:42:59.949204 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:42:59.949083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:39747)
0809 16:42:59.949150 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 16:42:59.949182 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 16:42:59.949424 15929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 16:42:59.963989 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:42:59.964248 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:42:59.964289 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:42:59.965427 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:42:59.965500 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:42:59.965546 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:42:59.967097 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:36961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=33651
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:42:59.967386 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:42:59.967628 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:42:59.972990 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:59.973075 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:42:59.973098 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:42:59.973800 15934 server_base.cc:1034] running on GCE node
I20260809 16:42:59.974083 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:42:59.974375 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:42:59.975564 15934 hybrid_clock.cc:648] HybridClock initialized: now 1786293779975528 us; error 47 us; skew 500 ppm
Aug 09 16:42:59 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293779, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:42:59.978421 15934 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:42:59.978981 15934 webserver.cc:460] Webserver started at http://127.28.62.130:33651/ using document root <none> and password file <none>
I20260809 16:42:59.979270 15934 fs_manager.cc:359] Metadata directory not provided
I20260809 16:42:59.979321 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:42:59.980814 15934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:42:59.981498 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:42:59.981683 15934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:42:59.981798 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "b510862baef34f859fb6d2f6f463c9f2"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "7be4ca700d1c48d2b7a22c47feebc8cd"
server_key_iv: "970764e26ab99731c0f9393f62769ee5"
server_key_version: "encryptionkey@0"
I20260809 16:42:59.982170 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:00.005525 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:00.008169 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:00.008378 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:00.008665 15934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:00.009013 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:00.009073 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:00.009258 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:00.009294 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:00.015918 15934 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36961
I20260809 16:43:00.015964 16086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36961 every 8 connection(s)
I20260809 16:43:00.016336 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:00.017609 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.017128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:58745)
0809 16:43:00.017327 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.017414 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
W20260809 16:43:00.018101 16088 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35167 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:35167: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
I20260809 16:43:00.018450 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.018315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:42349)
0809 16:43:00.018352 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.018420 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260809 16:43:00.018821 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.018718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:54409)
0809 16:43:00.018744 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.018800 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260809 16:43:00.018916 16088 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 16:43:00.019528 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 15934
I20260809 16:43:00.019882 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:32883
--local_ip_for_outbound_sockets=127.28.62.131
--tserver_master_addrs=127.28.62.190:35167
--webserver_port=35683
--webserver_interface=127.28.62.131
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:00.104856 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:00.105060 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:00.105084 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:00.106313 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:00.106359 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:00.106387 16093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:00.107879 16093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:32883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=35683
--tserver_master_addrs=127.28.62.190:35167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:00.108135 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:00.108407 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:00.113708 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:00.113696 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:00.113792 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:00.113837 16093 server_base.cc:1034] running on GCE node
I20260809 16:43:00.114137 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:00.114389 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:00.115530 16093 hybrid_clock.cc:648] HybridClock initialized: now 1786293780115508 us; error 38 us; skew 500 ppm
Aug 09 16:43:00 dist-test-slave-wl2h krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293780, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:00.118129 16093 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:43:00.118611 16093 webserver.cc:460] Webserver started at http://127.28.62.131:35683/ using document root <none> and password file <none>
I20260809 16:43:00.118861 16093 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:00.118933 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:00.120369 16093 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:00.121076 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:00.121341 16093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:00.121475 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "73761d49041b463aa59438092f4903e9"
format_stamp: "Formatted at 2026-08-09 16:42:18 on dist-test-slave-wl2h"
server_key: "e9e401744360f6eafbeec267875f6567"
server_key_iv: "1622c63315a8eac84e2ae60fec8f6e84"
server_key_version: "encryptionkey@0"
I20260809 16:43:00.121847 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:00.132131 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:00.134239 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:00.134409 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:00.134652 16093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:00.134964 16093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:00.135020 16093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:00.135150 16093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:00.135181 16093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:00.141160 16093 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:32883
I20260809 16:43:00.141212 16241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:32883 every 8 connection(s)
I20260809 16:43:00.141546 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:00.142613 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.142221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:35223)
0809 16:43:00.142385 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.142457 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260809 16:43:00.143069 16243 heartbeater.cc:644] Failed to heartbeat to 127.28.62.190:35167 (0 consecutive failures): Network error: Failed to ping master at 127.28.62.190:35167: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
I20260809 16:43:00.143359 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.143263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:33571)
0809 16:43:00.143313 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.143338 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 16:43:00.143672 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.143556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:42917)
0809 16:43:00.143630 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.143653 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 16:43:00.143791 16243 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 16:43:00.146361 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16093
I20260809 16:43:00.950158 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:00.949862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:33471)
0809 16:43:00.950024 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:00.950081 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260809 16:43:01.019465 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.019224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:33563)
0809 16:43:01.019383 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:01.019419 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260809 16:43:01.095818 16249 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:01.095948 16249 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:43:01.096369 16249 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883), b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
Aug 09 16:43:01 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293779, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.130@KRBTEST.COM
Aug 09 16:43:01 dist-test-slave-wl2h krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293779, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.131@KRBTEST.COM
I20260809 16:43:01.107743 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.096769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:54829 (local address 127.28.62.130:36961)
0809 16:43:01.096931 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:01.096945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:01.096972 (+ 27us) server_negotiation.cc:407] Connection header received
0809 16:43:01.097412 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:01.097414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:01.097464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:01.097574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:01.098463 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.099602 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.100684 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.101068 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.101181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:01.103666 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:01.103670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:01.103671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:01.103685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:01.105997 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106444 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:01.106447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:01.106447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:01.106548 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106811 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:01.106812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:01.106812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:01.107116 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:01.107210 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:01.107486 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 16:43:01.107610 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260809 16:43:01.107736 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.096593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36961 (local address 127.28.62.129:54829)
0809 16:43:01.096693 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:01.096708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:01.096786 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:01.097648 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:01.097651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:01.098050 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:01.098312 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:01.098319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.099694 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.099695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:01.100573 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:01.100575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.101170 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.101171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:01.101308 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:01.101984 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:01.101988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:01.103545 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:01.106100 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:01.106108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:01.106350 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:01.106628 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:01.106630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:01.106733 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:01.107191 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:01.107206 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:01.107323 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 16:43:01.107434 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 16:43:01.107620 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":4}
I20260809 16:43:01.107820 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.096770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:36381 (local address 127.28.62.131:32883)
0809 16:43:01.096918 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:01.096930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:01.096961 (+ 31us) server_negotiation.cc:407] Connection header received
0809 16:43:01.097447 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:01.097449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:01.097500 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:01.097668 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:01.098618 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.099814 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.100674 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.101071 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.101207 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:01.104300 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:01.104303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:01.104304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:01.104316 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:01.106244 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106608 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:01.106611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:01.106611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:01.106710 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106919 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:01.106920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:01.106920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:01.107226 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:01.107317 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:01.107526 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 16:43:01.107696 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260809 16:43:01.107856 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.096595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:32883 (local address 127.28.62.129:36381)
0809 16:43:01.096756 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:01.096767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:01.096825 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:01.097661 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:01.097662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:01.098051 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:01.098496 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:01.098499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.099913 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.099914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:01.100570 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:01.100572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:01.101179 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:01.101181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:01.101310 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:01.101922 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:01.101930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:01.103916 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:01.106338 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:01.106340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:01.106522 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:01.106791 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:01.106792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:01.106792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:01.106855 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:01.107311 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:01.107312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:01.107372 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 16:43:01.107458 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:43:01.107706 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":141,"thread_start_us":68,"threads_started":1}
W20260809 16:43:01.108791 15799 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
W20260809 16:43:01.108932 15800 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
I20260809 16:43:01.108997 15800 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1f7b3e65d0448b90e74a8674a0982d; no voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2
I20260809 16:43:01.109126 16249 raft_consensus.cc:2726] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:43:01.144330 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.144101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:38627)
0809 16:43:01.144250 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:01.144290 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260809 16:43:01.951195 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:01.950885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:47721)
0809 16:43:01.951100 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:01.951145 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260809 16:43:02.020291 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:02.020010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:53609)
0809 16:43:02.020182 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:02.020242 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260809 16:43:02.145217 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:02.144922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:37699)
0809 16:43:02.145103 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:02.145151 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260809 16:43:02.802819 16258 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:02.802960 16258 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:43:02.803216 16258 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883), b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
W20260809 16:43:02.803663 15800 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
W20260809 16:43:02.803761 15799 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
I20260809 16:43:02.803850 15799 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1f7b3e65d0448b90e74a8674a0982d; no voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2
I20260809 16:43:02.803947 16258 raft_consensus.cc:2726] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:43:02.952028 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:02.951785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:55625)
0809 16:43:02.951924 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:02.951975 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260809 16:43:03.021092 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:03.020847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:52935)
0809 16:43:03.021001 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:03.021048 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260809 16:43:03.145982 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:03.145718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:47731)
0809 16:43:03.145899 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:03.145938 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260809 16:43:03.952973 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:03.952712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:50927)
0809 16:43:03.952875 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:03.952919 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260809 16:43:04.021888 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:04.021633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:36215)
0809 16:43:04.021795 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:04.021838 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260809 16:43:04.146719 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:04.146505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:60119)
0809 16:43:04.146635 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:04.146676 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260809 16:43:04.418516 16267 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:04.418624 16267 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:43:04.418862 16267 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883), b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
W20260809 16:43:04.419273 15800 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
W20260809 16:43:04.419359 15799 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
I20260809 16:43:04.419435 15799 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1f7b3e65d0448b90e74a8674a0982d; no voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2
I20260809 16:43:04.419533 16267 raft_consensus.cc:2726] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:43:04.953871 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:04.953593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:47185)
0809 16:43:04.953777 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:04.953820 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260809 16:43:05.022776 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:05.022476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:53477)
0809 16:43:05.022677 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:05.022727 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260809 16:43:05.147478 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:05.147273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:60621)
0809 16:43:05.147400 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:05.147435 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260809 16:43:05.954807 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:05.954524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:51289)
0809 16:43:05.954707 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:05.954755 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260809 16:43:06.023552 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:06.023292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:36109)
0809 16:43:06.023462 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:06.023503 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260809 16:43:06.148352 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:06.148098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:54749)
0809 16:43:06.148259 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:06.148298 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260809 16:43:06.855563 16274 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:06.855671 16274 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:43:06.855841 16274 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883), b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
W20260809 16:43:06.856339 15799 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
W20260809 16:43:06.856426 15800 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
I20260809 16:43:06.856459 15800 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1f7b3e65d0448b90e74a8674a0982d; no voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2
I20260809 16:43:06.856536 16274 raft_consensus.cc:2726] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:43:06.955775 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:06.955505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:54707)
0809 16:43:06.955672 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:06.955722 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260809 16:43:07.024405 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:07.024099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:47859)
0809 16:43:07.024242 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:07.024285 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260809 16:43:07.149204 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:07.148903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:52383)
0809 16:43:07.149100 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:07.149153 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260809 16:43:07.956615 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:07.956371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:45805)
0809 16:43:07.956525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:07.956565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260809 16:43:08.025308 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:08.024979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:37059)
0809 16:43:08.025209 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:08.025261 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260809 16:43:08.149973 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:08.149754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:35205)
0809 16:43:08.149884 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:08.149924 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260809 16:43:08.957412 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:08.957158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:55481)
0809 16:43:08.957314 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:08.957359 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260809 16:43:09.026113 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:09.025862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:37095)
0809 16:43:09.026018 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:09.026060 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260809 16:43:09.150755 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:09.150520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:58995)
0809 16:43:09.150666 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:09.150706 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260809 16:43:09.958249 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:09.958020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:58131)
0809 16:43:09.958166 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:09.958201 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260809 16:43:10.027134 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:10.026718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:33241)
0809 16:43:10.026917 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:10.026966 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260809 16:43:10.151657 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:10.151414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:55801)
0809 16:43:10.151566 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:10.151605 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260809 16:43:10.959144 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:10.958824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:47239)
0809 16:43:10.958979 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:10.959073 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260809 16:43:11.027861 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:11.027640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:48687)
0809 16:43:11.027778 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:11.027813 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260809 16:43:11.152616 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:11.152307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:37813)
0809 16:43:11.152514 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:11.152563 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260809 16:43:11.659839 16293 raft_consensus.cc:482] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:11.659943 16293 raft_consensus.cc:504] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73761d49041b463aa59438092f4903e9" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 32883 } } peers { permanent_uuid: "b510862baef34f859fb6d2f6f463c9f2" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36961 } } peers { permanent_uuid: "8c1f7b3e65d0448b90e74a8674a0982d" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 42057 } }
I20260809 16:43:11.660221 16293 leader_election.cc:288] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883), b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961)
W20260809 16:43:11.660696 15799 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73761d49041b463aa59438092f4903e9 (127.28.62.131:32883): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
W20260809 16:43:11.660799 15800 leader_election.cc:341] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b510862baef34f859fb6d2f6f463c9f2 (127.28.62.130:36961): Not found: Tablet not found: f250c3e96a284de5ac04de967b61f7f3
I20260809 16:43:11.660867 15800 leader_election.cc:302] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1f7b3e65d0448b90e74a8674a0982d; no voters: 73761d49041b463aa59438092f4903e9, b510862baef34f859fb6d2f6f463c9f2
I20260809 16:43:11.660964 16293 raft_consensus.cc:2726] T f250c3e96a284de5ac04de967b61f7f3 P 8c1f7b3e65d0448b90e74a8674a0982d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 16:43:11.959960 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:11.959674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:43331)
0809 16:43:11.959838 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:11.959886 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260809 16:43:12.028661 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:12.028371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:34195)
0809 16:43:12.028548 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:12.028593 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260809 16:43:12.153311 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:12.153089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:41961)
0809 16:43:12.153219 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:12.153259 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260809 16:43:12.960806 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:12.960523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:50997)
0809 16:43:12.960691 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:12.960753 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260809 16:43:13.029486 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:13.029269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:41843)
0809 16:43:13.029401 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:13.029436 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260809 16:43:13.154196 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:13.153923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:34587)
0809 16:43:13.154102 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:13.154146 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260809 16:43:13.961550 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:13.961293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:39857)
0809 16:43:13.961452 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:13.961490 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260809 16:43:14.030321 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:14.030018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:46339)
0809 16:43:14.030207 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:14.030254 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260809 16:43:14.154959 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:14.154730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:37043)
0809 16:43:14.154868 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:14.154907 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260809 16:43:14.962504 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:14.962227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.129:50141)
0809 16:43:14.962403 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:14.962453 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260809 16:43:15.031286 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:15.030944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.130:38913)
0809 16:43:15.031176 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:15.031219 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":155,"threads_started":1}
I20260809 16:43:15.155453 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15735
I20260809 16:43:15.158179 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:15.155579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:35167 (local address 127.28.62.131:36689)
0809 16:43:15.158074 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:15.158121 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.190:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2470,"thread_start_us":64,"threads_started":1}
I20260809 16:43:15.165243 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 15934
I20260809 16:43:15.172798 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16093
2026-08-09T16:43:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58590 ms)
[----------] 30 tests from SecurityITest (239387 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:15 dist-test-slave-wl2h krb5kdc[16309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:15 dist-test-slave-wl2h krb5kdc[16309](info): set up 2 sockets
Aug 09 16:43:15 dist-test-slave-wl2h krb5kdc[16309](info): commencing operation
krb5kdc: starting...
W20260809 16:43:16.309315 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:16.438076 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:43037
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:16.520036 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:16.520314 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:16.520354 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:16.521514 16324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:16.521608 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:16.521634 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:16.521663 16324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:16.521704 16324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:16.523239 16324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:16.523531 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:16.523774 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:16.529330 16332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:16.529347 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:16.529304 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:16.529426 16324 server_base.cc:1034] running on GCE node
I20260809 16:43:16.529680 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:16.529928 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:16.531069 16324 hybrid_clock.cc:648] HybridClock initialized: now 1786293796531056 us; error 44 us; skew 500 ppm
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:16.537631 16324 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:16.538147 16324 webserver.cc:460] Webserver started at http://127.28.62.190:44987/ using document root <none> and password file <none>
I20260809 16:43:16.538404 16324 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:16.538475 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:16.538626 16324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:16.539783 16324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "59674608cc3b479cbc4924085f10639e"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "06195384cac6b854c61e7f11ac546423"
server_key_iv: "b6b4806f80fade25609583c39cf3f2b9"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.540154 16324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "59674608cc3b479cbc4924085f10639e"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "06195384cac6b854c61e7f11ac546423"
server_key_iv: "b6b4806f80fade25609583c39cf3f2b9"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.541550 16324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:16.542356 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.542595 16324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:16.542706 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "59674608cc3b479cbc4924085f10639e"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "06195384cac6b854c61e7f11ac546423"
server_key_iv: "b6b4806f80fade25609583c39cf3f2b9"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.542793 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:16.555521 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:16.557792 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:16.558063 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:16.562444 16324 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:43037
I20260809 16:43:16.562484 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:43037 every 8 connection(s)
I20260809 16:43:16.562879 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:16.563596 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16324
I20260809 16:43:16.563699 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:16.563912 28922 external_mini_cluster.cc:1428] Setting key 2c3379aee0ec927eec34553b867e4e09
I20260809 16:43:16.564061 16405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:16.567538 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: Bootstrap starting.
I20260809 16:43:16.568271 16405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:16.568606 16405 log.cc:824] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:16.569409 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: No bootstrap required, opened a new log
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:16.571683 16405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } }
I20260809 16:43:16.571873 16405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:16.571916 16405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59674608cc3b479cbc4924085f10639e, State: Initialized, Role: FOLLOWER
I20260809 16:43:16.572069 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } }
I20260809 16:43:16.572157 16405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:16.572180 16405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:16.572259 16405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:16.572974 16405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } }
I20260809 16:43:16.573067 16405 leader_election.cc:302] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59674608cc3b479cbc4924085f10639e; no voters:
I20260809 16:43:16.573264 16405 leader_election.cc:288] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:16.573294 16411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:16.573415 16411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [term 1 LEADER]: Becoming Leader. State: Replica: 59674608cc3b479cbc4924085f10639e, State: Running, Role: LEADER
I20260809 16:43:16.573562 16411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } }
I20260809 16:43:16.573549 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:16.573990 16414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59674608cc3b479cbc4924085f10639e. Latest consensus state: current_term: 1 leader_uuid: "59674608cc3b479cbc4924085f10639e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } } }
I20260809 16:43:16.574002 16413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59674608cc3b479cbc4924085f10639e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59674608cc3b479cbc4924085f10639e" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 43037 } } }
I20260809 16:43:16.574086 16414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:16.574107 16413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:16.574297 16416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:16.574996 16416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:16.576318 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:16.564498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.28.62.190:43037)
0809 16:43:16.564693 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:16.564702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:16.564741 (+ 39us) server_negotiation.cc:407] Connection header received
0809 16:43:16.565274 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:16.565284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:16.565332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:16.565444 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:16.566288 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.567232 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.568300 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.568800 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.568920 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:16.571537 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:16.571547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:16.571547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:16.571559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:16.573538 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.574560 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.574563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.574564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.574673 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.574940 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.574941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.574941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.575306 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:16.575908 (+ 602us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:16.575982 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 16:43:16.576119 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260809 16:43:16.577009 16416 catalog_manager.cc:1347] Generated new cluster ID: 81d2d72152ea478cab4126c6e629a161
I20260809 16:43:16.577062 16416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:16.593586 16416 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:16.594344 16416 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:16.602174 16416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: Generated new TSK 0
I20260809 16:43:16.602388 16416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:43:16.640465 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:16.741472 16437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:16.741693 16437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:16.741717 16437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:16.742863 16437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:16.742905 16437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:16.742933 16437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:16.744447 16437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:16.744707 16437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:16.744961 16437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:43:16.750648 16437 server_base.cc:1034] running on GCE node
W20260809 16:43:16.750609 16448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:16.750626 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:16.750649 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:16.751107 16437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:16.751361 16437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:16.752561 16437 hybrid_clock.cc:648] HybridClock initialized: now 1786293796752560 us; error 44 us; skew 500 ppm
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:16.755518 16437 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:43:16.756006 16437 webserver.cc:460] Webserver started at http://127.28.62.129:43133/ using document root <none> and password file <none>
I20260809 16:43:16.756219 16437 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:16.756264 16437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:16.756361 16437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:16.757529 16437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "8fe96c3e40d14b6cb931811700449b31"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "815b011a4d49947463346ae6923e4a02"
server_key_iv: "21896464d8b5ad059d76b0cadb05220c"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.757879 16437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "8fe96c3e40d14b6cb931811700449b31"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "815b011a4d49947463346ae6923e4a02"
server_key_iv: "21896464d8b5ad059d76b0cadb05220c"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.759377 16437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:16.760157 16456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.760351 16437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:43:16.760433 16437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "8fe96c3e40d14b6cb931811700449b31"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "815b011a4d49947463346ae6923e4a02"
server_key_iv: "21896464d8b5ad059d76b0cadb05220c"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.760488 16437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:16.777529 16437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:16.779991 16437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:16.780112 16437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:16.780323 16437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:16.780650 16437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:16.780687 16437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.780710 16437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:16.780723 16437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.786778 16437 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:38479
I20260809 16:43:16.786901 16588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:38479 every 8 connection(s)
I20260809 16:43:16.787223 16437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:43:16.787856 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16437
I20260809 16:43:16.787977 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:16.788208 28922 external_mini_cluster.cc:1428] Setting key ab712b306763be5e491e40ccb8146028
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
I20260809 16:43:16.799230 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:16.788042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:48057 (local address 127.28.62.190:43037)
0809 16:43:16.788106 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:16.788109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:16.788341 (+ 232us) server_negotiation.cc:407] Connection header received
0809 16:43:16.788952 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:16.788952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:16.788961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:16.789035 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:16.790029 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.790813 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.792462 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.792715 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.792786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:16.795556 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:16.795558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:16.795558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:16.795567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:16.797442 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.798024 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.798025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.798026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.798095 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.798408 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.798409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.798410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.798544 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:16.798635 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:16.798985 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 16:43:16.799125 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:43:16.799431 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:16.788026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:43037 (local address 127.28.62.129:48057)
0809 16:43:16.788218 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:16.788261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:16.788837 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:16.789120 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:16.789124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:16.789726 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:16.789924 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:16.789929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.790943 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.790944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:16.792352 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:16.792356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.792825 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.792826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:16.792980 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:16.793605 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:16.793615 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:16.795443 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:16.797575 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:16.797585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:16.797588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:16.797937 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:16.798191 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:16.798192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:16.798192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:16.798327 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:16.798667 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:16.798674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:16.798826 (+ 152us) client_negotiation.cc:769] Sending connection context
0809 16:43:16.798946 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 16:43:16.799162 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260809 16:43:16.800122 16589 heartbeater.cc:347] Connected to a master server at 127.28.62.190:43037
I20260809 16:43:16.800254 16589 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:16.800448 16589 heartbeater.cc:511] Master 127.28.62.190:43037 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
I20260809 16:43:16.801051 16361 ts_manager.cc:194] Registered new tserver with Master: 8fe96c3e40d14b6cb931811700449b31 (127.28.62.129:38479)
I20260809 16:43:16.801728 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:48057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:43:16.818872 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:16.910274 16595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:16.910516 16595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:16.910553 16595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:16.911703 16595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:16.911773 16595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:16.911819 16595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:16.913334 16595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:16.913616 16595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:16.913863 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:16.919188 16603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:16.919247 16601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:16.919291 16595 server_base.cc:1034] running on GCE node
W20260809 16:43:16.919251 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:16.919706 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:16.919945 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:16.921108 16595 hybrid_clock.cc:648] HybridClock initialized: now 1786293796921075 us; error 39 us; skew 500 ppm
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:16.923731 16595 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:43:16.924233 16595 webserver.cc:460] Webserver started at http://127.28.62.130:39323/ using document root <none> and password file <none>
I20260809 16:43:16.924485 16595 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:16.924557 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:16.924712 16595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:16.925906 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "35fdea82849945a4a89524ee5d0c5bb4"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "7fd39f2c406fefda41861cc3b91bc4e3"
server_key_iv: "0631451772ba79fe5d90bb8aeed74b64"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.926275 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "35fdea82849945a4a89524ee5d0c5bb4"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "7fd39f2c406fefda41861cc3b91bc4e3"
server_key_iv: "0631451772ba79fe5d90bb8aeed74b64"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.927785 16595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:16.928615 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.928833 16595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:16.928946 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "35fdea82849945a4a89524ee5d0c5bb4"
format_stamp: "Formatted at 2026-08-09 16:43:16 on dist-test-slave-wl2h"
server_key: "7fd39f2c406fefda41861cc3b91bc4e3"
server_key_iv: "0631451772ba79fe5d90bb8aeed74b64"
server_key_version: "encryptionkey@0"
I20260809 16:43:16.929029 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:16.941592 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:16.943542 16595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:16.943720 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:16.943971 16595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:16.944285 16595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:16.944341 16595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.944386 16595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:16.944428 16595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:16.951166 16595 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:35569
I20260809 16:43:16.951251 16741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:35569 every 8 connection(s)
I20260809 16:43:16.951604 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:16.955358 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16595
I20260809 16:43:16.955487 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:16.955673 28922 external_mini_cluster.cc:1428] Setting key 55f9b5066a45c5f06bac36e99331eec9
Aug 09 16:43:16 dist-test-slave-wl2h krb5kdc[16309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293796, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:16.962826 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:16.952435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:51037 (local address 127.28.62.190:43037)
0809 16:43:16.952481 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:16.952483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:16.952700 (+ 217us) server_negotiation.cc:407] Connection header received
0809 16:43:16.953194 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:16.953195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:16.953205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:16.953252 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:16.954235 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.954726 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.956176 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.956474 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.956544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:16.959538 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:16.959539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:16.959539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:16.959547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:16.961325 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.961760 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.961762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.961762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.961829 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:16.962092 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:16.962092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:16.962093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:16.962250 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:16.962328 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:16.962578 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 16:43:16.962676 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:16.962844 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:16.952379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:43037 (local address 127.28.62.130:51037)
0809 16:43:16.952565 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:16.952611 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:16.953084 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:16.953339 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:16.953342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:16.953914 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:16.954102 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:16.954107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.954819 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.954820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:16.956091 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:16.956094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:16.956583 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:16.956584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:16.956746 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:16.957342 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:16.957352 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:16.959410 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:16.961423 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:16.961430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:16.961432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:16.961674 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:16.961914 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:16.961914 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:16.961915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:16.962024 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:16.962321 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:16.962326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:16.962444 (+ 118us) client_negotiation.cc:769] Sending connection context
0809 16:43:16.962528 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 16:43:16.962657 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260809 16:43:16.963553 16742 heartbeater.cc:347] Connected to a master server at 127.28.62.190:43037
I20260809 16:43:16.963670 16742 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:16.963900 16742 heartbeater.cc:511] Master 127.28.62.190:43037 requested a full tablet report, sending...
I20260809 16:43:16.964370 16361 ts_manager.cc:194] Registered new tserver with Master: 35fdea82849945a4a89524ee5d0c5bb4 (127.28.62.130:35569)
I20260809 16:43:16.964779 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:51037
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:43:16.990038 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:17.074018 16748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:17.074231 16748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:17.074258 16748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:17.075618 16748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:17.075661 16748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:17.075690 16748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:17.077147 16748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:17.077396 16748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:17.077636 16748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:17.082825 16755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.082854 16754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.082950 16748 server_base.cc:1034] running on GCE node
W20260809 16:43:17.082825 16757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.083209 16748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:17.083424 16748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:17.084563 16748 hybrid_clock.cc:648] HybridClock initialized: now 1786293797084542 us; error 39 us; skew 500 ppm
Aug 09 16:43:17 dist-test-slave-wl2h krb5kdc[16309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:17.087050 16748 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:43:17.087459 16748 webserver.cc:460] Webserver started at http://127.28.62.131:39207/ using document root <none> and password file <none>
I20260809 16:43:17.087643 16748 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:17.087682 16748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:17.087780 16748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:17.088845 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "06ac53f9262c45088c42a1398a2d9a08"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "649f77729ec5ee0cc1fee13c7dc44cde"
server_key_iv: "4c29a9efb56f72193f59deba72f8178f"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.089149 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "06ac53f9262c45088c42a1398a2d9a08"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "649f77729ec5ee0cc1fee13c7dc44cde"
server_key_iv: "4c29a9efb56f72193f59deba72f8178f"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.090404 16748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:43:17.091156 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.091313 16748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:17.091403 16748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "06ac53f9262c45088c42a1398a2d9a08"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "649f77729ec5ee0cc1fee13c7dc44cde"
server_key_iv: "4c29a9efb56f72193f59deba72f8178f"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.091455 16748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:17.102324 16748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:17.104700 16748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:17.104871 16748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:17.105105 16748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:17.105441 16748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:17.105502 16748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.105544 16748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:17.105572 16748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.111490 16748 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:44681
I20260809 16:43:17.111562 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:44681 every 8 connection(s)
I20260809 16:43:17.111896 16748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:17.116125 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16748
I20260809 16:43:17.116242 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:17.116439 28922 external_mini_cluster.cc:1428] Setting key 4eb55d58b4efc426ebd4cb1657ee66f4
Aug 09 16:43:17 dist-test-slave-wl2h krb5kdc[16309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:17.122241 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.112655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:35271 (local address 127.28.62.190:43037)
0809 16:43:17.112718 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.112720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.112947 (+ 227us) server_negotiation.cc:407] Connection header received
0809 16:43:17.113433 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.113434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.113443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.113502 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.114417 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.114908 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.115969 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.116287 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.116368 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.118787 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.118788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.118789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:17.118796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.120567 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:17.121096 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:17.121097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:17.121097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:17.121160 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:17.121512 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:17.121514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:17.121514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:17.121643 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.121712 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.121985 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.122078 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 16:43:17.122283 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.112635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:43037 (local address 127.28.62.131:35271)
0809 16:43:17.112809 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.112853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.113349 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.113551 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.113554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.114132 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:17.114330 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.114334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.114995 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.114996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.115861 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.115864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.116391 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.116392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.116533 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.117155 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:17.117166 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:17.118670 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:17.120682 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:17.120689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:17.120691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:17.120981 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:17.121273 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:17.121274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:17.121274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:17.121407 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:17.121721 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:17.121726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:17.121858 (+ 132us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.121948 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.122083 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260809 16:43:17.123076 16900 heartbeater.cc:347] Connected to a master server at 127.28.62.190:43037
I20260809 16:43:17.123219 16900 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:17.123440 16900 heartbeater.cc:511] Master 127.28.62.190:43037 requested a full tablet report, sending...
I20260809 16:43:17.123852 16361 ts_manager.cc:194] Registered new tserver with Master: 06ac53f9262c45088c42a1398a2d9a08 (127.28.62.131:44681)
I20260809 16:43:17.124198 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:35271
I20260809 16:43:17.128230 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:17.136808 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.129710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.28.62.190:43037)
0809 16:43:17.129787 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.129789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.129878 (+ 89us) server_negotiation.cc:407] Connection header received
0809 16:43:17.129947 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.129948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.129958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.130028 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.130652 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.131216 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.132144 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.132361 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.132452 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.133918 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.133919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.133919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:17.133927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.135548 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:17.135919 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:17.135920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:17.135921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:17.135988 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:17.136195 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:17.136196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:17.136196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:17.136362 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.136482 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.136615 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.136700 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:43:17.138717 16361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:17.140472 16361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:17.148283 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38479 (local address 127.0.0.1:46414)
0809 16:43:17.144364 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.144374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.144427 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.144678 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.144679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.144682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.144753 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.144755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.146014 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.146015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.147258 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.147261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.147989 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.147990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.148074 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.148079 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.148145 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.148195 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":123,"threads_started":1}
I20260809 16:43:17.148296 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.28.62.129:38479)
0809 16:43:17.144234 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.144239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.144494 (+ 255us) server_negotiation.cc:407] Connection header received
0809 16:43:17.144542 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.144544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.144560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.144681 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.144848 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.145882 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.147364 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.147882 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.147985 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.148052 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.148162 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.148206 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260809 16:43:17.148540 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:44681 (local address 127.0.0.1:57338)
0809 16:43:17.143512 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.143556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.143647 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.144248 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.144250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.144253 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.144334 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.144336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.145448 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.145449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.146815 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.146819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.147893 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.147894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.147981 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.147996 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.148059 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.148112 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260809 16:43:17.148846 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.28.62.131:44681)
0809 16:43:17.144023 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.144030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.144041 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:43:17.144088 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.144090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.144108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.144237 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.144449 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.145324 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.146920 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.147778 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.148623 (+ 845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.148689 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.148717 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.148758 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":99,"threads_started":1}
I20260809 16:43:17.149628 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35569 (local address 127.0.0.1:42344)
0809 16:43:17.144031 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.144043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.144118 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.145478 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.145478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.145481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.145541 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.145542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.147083 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.147084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.148257 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.148260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.149330 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.149331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.149408 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.149413 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.149480 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.149523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":122,"threads_started":1}
I20260809 16:43:17.149679 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.143812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.28.62.130:35569)
0809 16:43:17.144241 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.144245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.144257 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:17.144300 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.144302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.144319 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.144433 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.145638 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.146948 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.148360 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.149162 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.149236 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.149292 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.149530 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.149586 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":96,"threads_started":1}
I20260809 16:43:17.149576 16512 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d (DEFAULT_TABLE table=test-table [id=5240931044774dbaa513e409c6b4e808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.149574 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d (DEFAULT_TABLE table=test-table [id=5240931044774dbaa513e409c6b4e808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.149966 16512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef2532806cb4796bbe8954cff15fe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.149971 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef2532806cb4796bbe8954cff15fe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.150779 16669 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d (DEFAULT_TABLE table=test-table [id=5240931044774dbaa513e409c6b4e808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.151157 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef2532806cb4796bbe8954cff15fe1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.153146 16920 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Bootstrap starting.
I20260809 16:43:17.153880 16921 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: Bootstrap starting.
I20260809 16:43:17.153940 16920 tablet_bootstrap.cc:654] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.154485 16920 log.cc:824] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.154673 16921 tablet_bootstrap.cc:654] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.154846 16922 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Bootstrap starting.
I20260809 16:43:17.155413 16921 log.cc:824] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.155723 16920 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: No bootstrap required, opened a new log
I20260809 16:43:17.155728 16922 tablet_bootstrap.cc:654] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.155840 16920 ts_tablet_manager.cc:1397] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:43:17.156286 16921 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: No bootstrap required, opened a new log
I20260809 16:43:17.156304 16922 log.cc:824] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.156368 16921 ts_tablet_manager.cc:1397] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:43:17.157150 16922 tablet_bootstrap.cc:492] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: No bootstrap required, opened a new log
I20260809 16:43:17.157246 16922 ts_tablet_manager.cc:1397] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:17.158093 16920 raft_consensus.cc:348] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.158305 16920 raft_consensus.cc:374] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.158347 16920 raft_consensus.cc:729] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ac53f9262c45088c42a1398a2d9a08, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.158552 16920 consensus_queue.cc:260] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.158555 16921 raft_consensus.cc:348] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.158723 16921 raft_consensus.cc:374] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.158809 16921 raft_consensus.cc:729] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe96c3e40d14b6cb931811700449b31, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.158921 16920 ts_tablet_manager.cc:1428] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:17.158955 16900 heartbeater.cc:503] Master 127.28.62.190:43037 was elected leader, sending a full tablet report...
I20260809 16:43:17.159106 16921 consensus_queue.cc:260] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.159065 16922 raft_consensus.cc:348] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.159232 16922 raft_consensus.cc:374] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.159273 16922 raft_consensus.cc:729] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fdea82849945a4a89524ee5d0c5bb4, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.159405 16922 consensus_queue.cc:260] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.159435 16589 heartbeater.cc:503] Master 127.28.62.190:43037 was elected leader, sending a full tablet report...
I20260809 16:43:17.159719 16922 ts_tablet_manager.cc:1428] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:43:17.159776 16742 heartbeater.cc:503] Master 127.28.62.190:43037 was elected leader, sending a full tablet report...
I20260809 16:43:17.159422 16921 ts_tablet_manager.cc:1428] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:17.198148 16926 raft_consensus.cc:482] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:17.198347 16926 raft_consensus.cc:504] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.198805 16926 leader_election.cc:288] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fe96c3e40d14b6cb931811700449b31 (127.28.62.129:38479), 35fdea82849945a4a89524ee5d0c5bb4 (127.28.62.130:35569)
W20260809 16:43:17.202198 16743 tablet.cc:2369] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:17.202935 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.199115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:35533 (local address 127.28.62.130:35569)
0809 16:43:17.199163 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.199165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.199467 (+ 302us) server_negotiation.cc:407] Connection header received
0809 16:43:17.199506 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.199506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.199538 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.199595 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.199772 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.200449 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.201933 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.202534 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.202600 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.202609 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.202827 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.202867 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 16:43:17.202935 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.199161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:39585 (local address 127.28.62.129:38479)
0809 16:43:17.199227 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.199229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.199239 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:17.199271 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.199272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.199283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.199356 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.199560 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.200335 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.201934 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.202517 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.202600 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.202625 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.202825 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.202868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 16:43:17.202958 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.198988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38479 (local address 127.28.62.131:39585)
0809 16:43:17.199100 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.199113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.199165 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.199385 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.199386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.199388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.199472 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.199474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.200425 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.200426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.201828 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.201831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.202633 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.202634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.202736 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.202738 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.202803 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.202855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":15}
I20260809 16:43:17.202970 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.199083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35569 (local address 127.28.62.131:35533)
0809 16:43:17.199373 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.199384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.199453 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.199637 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.199638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.199639 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.199693 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.199694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.200528 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.200528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.201827 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.201831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.202633 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.202634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.202727 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.202731 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.202803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.202855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":15,"thread_start_us":185,"threads_started":1}
I20260809 16:43:17.203615 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef2532806cb4796bbe8954cff15fe1d" candidate_uuid: "06ac53f9262c45088c42a1398a2d9a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe96c3e40d14b6cb931811700449b31" is_pre_election: true
I20260809 16:43:17.203620 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef2532806cb4796bbe8954cff15fe1d" candidate_uuid: "06ac53f9262c45088c42a1398a2d9a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fdea82849945a4a89524ee5d0c5bb4" is_pre_election: true
I20260809 16:43:17.203781 16536 raft_consensus.cc:2454] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ac53f9262c45088c42a1398a2d9a08 in term 0.
I20260809 16:43:17.203781 16691 raft_consensus.cc:2454] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ac53f9262c45088c42a1398a2d9a08 in term 0.
I20260809 16:43:17.204034 16769 leader_election.cc:302] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ac53f9262c45088c42a1398a2d9a08, 35fdea82849945a4a89524ee5d0c5bb4; no voters:
I20260809 16:43:17.204186 16926 raft_consensus.cc:2781] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:17.204249 16926 raft_consensus.cc:482] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:17.204275 16926 raft_consensus.cc:3037] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:17.205080 16926 raft_consensus.cc:504] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.205242 16926 leader_election.cc:288] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [CANDIDATE]: Term 1 election: Requested vote from peers 8fe96c3e40d14b6cb931811700449b31 (127.28.62.129:38479), 35fdea82849945a4a89524ee5d0c5bb4 (127.28.62.130:35569)
I20260809 16:43:17.205379 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef2532806cb4796bbe8954cff15fe1d" candidate_uuid: "06ac53f9262c45088c42a1398a2d9a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe96c3e40d14b6cb931811700449b31"
I20260809 16:43:17.205440 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef2532806cb4796bbe8954cff15fe1d" candidate_uuid: "06ac53f9262c45088c42a1398a2d9a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fdea82849945a4a89524ee5d0c5bb4"
I20260809 16:43:17.205487 16536 raft_consensus.cc:3037] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:17.205551 16691 raft_consensus.cc:3037] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:17.206308 16536 raft_consensus.cc:2454] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ac53f9262c45088c42a1398a2d9a08 in term 1.
I20260809 16:43:17.206308 16691 raft_consensus.cc:2454] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ac53f9262c45088c42a1398a2d9a08 in term 1.
I20260809 16:43:17.206504 16767 leader_election.cc:302] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ac53f9262c45088c42a1398a2d9a08, 8fe96c3e40d14b6cb931811700449b31; no voters:
I20260809 16:43:17.206645 16926 raft_consensus.cc:2781] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:17.206859 16926 raft_consensus.cc:686] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 1 LEADER]: Becoming Leader. State: Replica: 06ac53f9262c45088c42a1398a2d9a08, State: Running, Role: LEADER
I20260809 16:43:17.206969 16926 consensus_queue.cc:237] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } }
I20260809 16:43:17.207973 16360 catalog_manager.cc:5495] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ac53f9262c45088c42a1398a2d9a08 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "06ac53f9262c45088c42a1398a2d9a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ac53f9262c45088c42a1398a2d9a08" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 44681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 } health_report { overall_health: UNKNOWN } } }
I20260809 16:43:17.223105 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.220737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.28.62.131:44681)
0809 16:43:17.220810 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.220812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.220835 (+ 23us) server_negotiation.cc:407] Connection header received
0809 16:43:17.220870 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.220871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.220888 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.220945 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:17.221061 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.221525 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.222331 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.222512 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.222565 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.222703 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:17.222796 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:17.222857 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.222926 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.222942 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:43:17.225014 16536 raft_consensus.cc:1261] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31 [term 1 FOLLOWER]: Refusing update from remote peer 06ac53f9262c45088c42a1398a2d9a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:17.225037 16691 raft_consensus.cc:1261] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 1 FOLLOWER]: Refusing update from remote peer 06ac53f9262c45088c42a1398a2d9a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:17.225433 16930 consensus_queue.cc:1035] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe96c3e40d14b6cb931811700449b31" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:17.225574 16926 consensus_queue.cc:1035] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fdea82849945a4a89524ee5d0c5bb4" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:17.228565 16934 mvcc.cc:205] Tried to move back new op lower bound from 7316659393430855680 to 7316659393360728064. Current Snapshot: MvccSnapshot[applied={T|T < 7316659393430855680}]
I20260809 16:43:17.231003 16360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:17.231146 16360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:17.231962 16360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59674608cc3b479cbc4924085f10639e: Sending DeleteTablet for 3 replicas of tablet 5ef2532806cb4796bbe8954cff15fe1d
I20260809 16:43:17.232359 16669 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:17 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:42344
I20260809 16:43:17.232369 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:17 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:57338
I20260809 16:43:17.232391 16512 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ef2532806cb4796bbe8954cff15fe1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:17 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:46414
I20260809 16:43:17.232599 16944 tablet_replica.cc:330] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: stopping tablet replica
I20260809 16:43:17.232671 16948 tablet_replica.cc:330] T 5ef2532806cb4796bbe8954cff15fe1d P 8fe96c3e40d14b6cb931811700449b31: stopping tablet replica
I20260809 16:43:17.232743 16944 raft_consensus.cc:2229] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:17.232762 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16437
I20260809 16:43:17.234642 16944 raft_consensus.cc:2258] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:17.236441 16944 ts_tablet_manager.cc:1905] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:17.238472 16944 ts_tablet_manager.cc:1918] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:17.238579 16944 log.cc:1196] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/5ef2532806cb4796bbe8954cff15fe1d
I20260809 16:43:17.238870 16944 ts_tablet_manager.cc:1939] T 5ef2532806cb4796bbe8954cff15fe1d P 06ac53f9262c45088c42a1398a2d9a08: Deleting consensus metadata
I20260809 16:43:17.239389 16341 catalog_manager.cc:4841] TS 06ac53f9262c45088c42a1398a2d9a08 (127.28.62.131:44681): tablet 5ef2532806cb4796bbe8954cff15fe1d (table test-table [id=5240931044774dbaa513e409c6b4e808]) successfully deleted
W20260809 16:43:17.241969 16341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:38479 (error 108)
I20260809 16:43:17.242092 16945 tablet_replica.cc:330] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4: stopping tablet replica
I20260809 16:43:17.242220 16945 raft_consensus.cc:2229] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:17.242421 16945 raft_consensus.cc:2258] T 5ef2532806cb4796bbe8954cff15fe1d P 35fdea82849945a4a89524ee5d0c5bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:17.242504 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.242385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38479 (local address 127.0.0.1:46426)
0809 16:43:17.242434 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.242465 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 16:43:17.242609 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16595
W20260809 16:43:17.242717 16341 catalog_manager.cc:4584] TS 8fe96c3e40d14b6cb931811700449b31 (127.28.62.129:38479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ef2532806cb4796bbe8954cff15fe1d: Network error: Client connection negotiation failed: client connection to 127.28.62.129:38479: connect: Connection refused (error 111)
I20260809 16:43:17.263614 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.263470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:35569 (local address 127.0.0.1:42352)
0809 16:43:17.263533 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.263563 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.130:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 16:43:17.263635 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16748
I20260809 16:43:17.272821 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16324
2026-08-09T16:43:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2099 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T16:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:17Z Disabled control of system clock
I20260809 16:43:17.303613 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:32793
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:32793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:17.389348 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:17.389570 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:17.389593 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:17.390678 16955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:17.390713 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:17.390726 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:17.390739 16955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:17.390751 16955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:17.392179 16955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:32793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:17.392393 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:17.392629 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:17.397619 16961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.397629 16966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.397727 16964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.397735 16955 server_base.cc:1034] running on GCE node
I20260809 16:43:17.398196 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:17.398509 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:17.399642 16955 hybrid_clock.cc:648] HybridClock initialized: now 1786293797399620 us; error 41 us; skew 500 ppm
I20260809 16:43:17.400893 16955 webserver.cc:460] Webserver started at http://127.28.62.190:40081/ using document root <none> and password file <none>
I20260809 16:43:17.401152 16955 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:17.401201 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:17.401372 16955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:17.402470 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "9e3cd0cb2b614c6ab409d0bd17174dac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "71d882a38e6ae11ef6d244cdf362d480"
server_key_iv: "9c75451597c08034f7bf6f836fe1b577"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.402849 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "9e3cd0cb2b614c6ab409d0bd17174dac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "71d882a38e6ae11ef6d244cdf362d480"
server_key_iv: "9c75451597c08034f7bf6f836fe1b577"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.404232 16955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:17.405051 16975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.405244 16955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:17.405380 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "9e3cd0cb2b614c6ab409d0bd17174dac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "71d882a38e6ae11ef6d244cdf362d480"
server_key_iv: "9c75451597c08034f7bf6f836fe1b577"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.405463 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:17.424331 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:17.424687 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:17.424857 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:17.429006 16955 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:32793
I20260809 16:43:17.429044 17039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:32793 every 8 connection(s)
I20260809 16:43:17.429400 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:17.430502 17040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.433022 17040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: Bootstrap starting.
I20260809 16:43:17.433600 17040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.433849 17040 log.cc:824] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.434505 17040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: No bootstrap required, opened a new log
I20260809 16:43:17.436125 17040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } }
I20260809 16:43:17.436261 17040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.436283 17040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3cd0cb2b614c6ab409d0bd17174dac, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.436436 17040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } }
I20260809 16:43:17.436520 17040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:17.436544 17040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:17.436571 17040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:17.437054 17040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } }
I20260809 16:43:17.437146 17040 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3cd0cb2b614c6ab409d0bd17174dac; no voters:
I20260809 16:43:17.437269 17040 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:17.437337 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:17.437491 17044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [term 1 LEADER]: Becoming Leader. State: Replica: 9e3cd0cb2b614c6ab409d0bd17174dac, State: Running, Role: LEADER
I20260809 16:43:17.437543 17040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:17.437601 17044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } }
I20260809 16:43:17.438052 17046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3cd0cb2b614c6ab409d0bd17174dac. Latest consensus state: current_term: 1 leader_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } } }
I20260809 16:43:17.438156 17046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:17.438139 17045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3cd0cb2b614c6ab409d0bd17174dac" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 32793 } } }
I20260809 16:43:17.438194 17045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:17.438410 17053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:17.438499 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 16955
I20260809 16:43:17.438612 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:17.438812 28922 external_mini_cluster.cc:1428] Setting key 5bf2a889a440cb34dcf86ee7d948feaa
I20260809 16:43:17.438943 17053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:17.440850 17053 catalog_manager.cc:1347] Generated new cluster ID: e79f2df27eaa4f76967303e25c5260bc
I20260809 16:43:17.440920 17053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:17.445324 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.439520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.28.62.190:32793)
0809 16:43:17.439722 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.439731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.439928 (+ 197us) server_negotiation.cc:407] Connection header received
0809 16:43:17.440070 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.440078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.440166 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.440329 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.440578 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.441931 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.443009 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.443665 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.443796 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.444554 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.444558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.444558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:17.444792 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.444840 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:17.444855 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.444929 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.445018 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.445102 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260809 16:43:17.457482 17053 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:17.458171 17053 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:17.462821 17053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: Generated new TSK 0
I20260809 16:43:17.463006 17053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:17.467293 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--builtin_ntp_servers=127.28.62.148:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:17.567279 17068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:17.567481 17068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:17.567504 17068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:17.568643 17068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:17.568686 17068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:17.568711 17068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:17.570039 17068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:17.570329 17068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:17.570657 17068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:17.576149 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.576267 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.576165 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.576573 17068 server_base.cc:1034] running on GCE node
I20260809 16:43:17.576817 17068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:17.577076 17068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:17.578284 17068 hybrid_clock.cc:648] HybridClock initialized: now 1786293797578265 us; error 31 us; skew 500 ppm
I20260809 16:43:17.579797 17068 webserver.cc:460] Webserver started at http://127.28.62.129:33881/ using document root <none> and password file <none>
I20260809 16:43:17.580089 17068 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:17.580180 17068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:17.580351 17068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:17.581516 17068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "57282f902cc64fb38dba8778076681c5"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "aa39151acbd2f2a0b402ef32aee88cd4"
server_key_iv: "3d7330865aca1f3986ca762608565459"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.581924 17068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "57282f902cc64fb38dba8778076681c5"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "aa39151acbd2f2a0b402ef32aee88cd4"
server_key_iv: "3d7330865aca1f3986ca762608565459"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.583448 17068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:17.584311 17084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.584530 17068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:17.584619 17068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "57282f902cc64fb38dba8778076681c5"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "aa39151acbd2f2a0b402ef32aee88cd4"
server_key_iv: "3d7330865aca1f3986ca762608565459"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.584686 17068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:17.597131 17068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:17.597445 17068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:17.597556 17068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:17.597741 17068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:17.598016 17068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:17.598045 17068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.598067 17068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:17.598079 17068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.603976 17068 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:43859
I20260809 16:43:17.604099 17219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:43859 every 8 connection(s)
I20260809 16:43:17.604419 17068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:43:17.610021 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.605270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:44361 (local address 127.28.62.190:32793)
0809 16:43:17.605361 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.605364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.605575 (+ 211us) server_negotiation.cc:407] Connection header received
0809 16:43:17.606047 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.606048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.606061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.606115 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.606771 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.607355 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.608533 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.608723 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.608786 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.609783 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.609784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.609784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:17.609788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.609803 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:17.609807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.609851 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.609934 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.609959 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 16:43:17.610270 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.605231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:32793 (local address 127.28.62.129:44361)
0809 16:43:17.605452 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.605486 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.605965 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.606179 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.606182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.606190 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:17.606690 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.606694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.607469 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.607470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.608453 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.608457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.608814 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.608815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.608929 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.609648 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:17.609649 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:17.609678 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:17.609701 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:17.609867 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:17.609871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:17.609874 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.609925 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.610006 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260809 16:43:17.610795 17220 heartbeater.cc:347] Connected to a master server at 127.28.62.190:32793
I20260809 16:43:17.610884 17220 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:17.611099 17220 heartbeater.cc:511] Master 127.28.62.190:32793 requested a full tablet report, sending...
I20260809 16:43:17.611613 16999 ts_manager.cc:194] Registered new tserver with Master: 57282f902cc64fb38dba8778076681c5 (127.28.62.129:43859)
I20260809 16:43:17.612267 16999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.129:44361
I20260809 16:43:17.614306 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17068
I20260809 16:43:17.614391 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:17.614545 28922 external_mini_cluster.cc:1428] Setting key 80133f30e1f8d88a9e28c51884c2a6fe
I20260809 16:43:17.615586 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--builtin_ntp_servers=127.28.62.148:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:17.699542 17223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:17.699756 17223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:17.699779 17223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:17.700840 17223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:17.700922 17223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:17.700969 17223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:17.702296 17223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:17.702486 17223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:17.702731 17223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:17.707789 17231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.707741 17228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.707892 17223 server_base.cc:1034] running on GCE node
W20260809 16:43:17.707743 17229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.708256 17223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:17.708520 17223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:17.709661 17223 hybrid_clock.cc:648] HybridClock initialized: now 1786293797709650 us; error 32 us; skew 500 ppm
I20260809 16:43:17.711081 17223 webserver.cc:460] Webserver started at http://127.28.62.130:32961/ using document root <none> and password file <none>
I20260809 16:43:17.711374 17223 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:17.711423 17223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:17.711603 17223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:17.712687 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "2198d4cca03d4c2b88ae6ef821a6eeac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "74594a1730656b9744364ff7b73f0bc4"
server_key_iv: "08d41bacf501920bf6de12e6c6284d85"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.713052 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "2198d4cca03d4c2b88ae6ef821a6eeac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "74594a1730656b9744364ff7b73f0bc4"
server_key_iv: "08d41bacf501920bf6de12e6c6284d85"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.714401 17223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:17.715153 17238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.715354 17223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:17.715456 17223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "2198d4cca03d4c2b88ae6ef821a6eeac"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "74594a1730656b9744364ff7b73f0bc4"
server_key_iv: "08d41bacf501920bf6de12e6c6284d85"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.715536 17223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:17.729249 17223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:17.729586 17223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:17.729753 17223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:17.730005 17223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:17.730378 17223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:17.730413 17223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.730437 17223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:17.730486 17223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.736951 17223 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:37173
I20260809 16:43:17.736996 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:37173 every 8 connection(s)
I20260809 16:43:17.737409 17223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:17.741618 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17223
I20260809 16:43:17.741707 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:17.741927 28922 external_mini_cluster.cc:1428] Setting key 5e73603d1a4f41bd6e1c65dd9d1521ee
I20260809 16:43:17.742956 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--builtin_ntp_servers=127.28.62.148:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 16:43:17.743294 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.738196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:59229 (local address 127.28.62.190:32793)
0809 16:43:17.738245 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.738261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.738607 (+ 346us) server_negotiation.cc:407] Connection header received
0809 16:43:17.739218 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.739220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.739232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.739303 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.740079 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.740590 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.741594 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.741892 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.741992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.742922 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.742923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.742924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:17.742928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.742942 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:17.742946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.742994 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.743184 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.743215 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:17.743672 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.738187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:32793 (local address 127.28.62.130:59229)
0809 16:43:17.738438 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.738510 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.739102 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.739374 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.739378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.739388 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:17.739976 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.739980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.740697 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.740698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.741499 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.741501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.742012 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.742013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.742152 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.742768 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:17.742769 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:17.742801 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:17.742823 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:17.742993 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:17.742998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:17.743001 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.743412 (+ 411us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.743520 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260809 16:43:17.744098 17365 heartbeater.cc:347] Connected to a master server at 127.28.62.190:32793
I20260809 16:43:17.744203 17365 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:17.744391 17365 heartbeater.cc:511] Master 127.28.62.190:32793 requested a full tablet report, sending...
I20260809 16:43:17.744762 16999 ts_manager.cc:194] Registered new tserver with Master: 2198d4cca03d4c2b88ae6ef821a6eeac (127.28.62.130:37173)
I20260809 16:43:17.745157 16999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.130:59229
W20260809 16:43:17.827068 17369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:17.827288 17369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:17.827311 17369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:17.828485 17369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:17.828531 17369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:17.828557 17369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:17.829908 17369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:17.830158 17369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:17.830489 17369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:17.836099 17377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.836107 17375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:17.836162 17374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:17.836193 17369 server_base.cc:1034] running on GCE node
I20260809 16:43:17.836642 17369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:17.836860 17369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:17.838001 17369 hybrid_clock.cc:648] HybridClock initialized: now 1786293797837989 us; error 31 us; skew 500 ppm
I20260809 16:43:17.839478 17369 webserver.cc:460] Webserver started at http://127.28.62.131:35111/ using document root <none> and password file <none>
I20260809 16:43:17.839758 17369 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:17.839810 17369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:17.839951 17369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:17.840994 17369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "3caba0e2eb3c47119e2813d66420e835"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "0a74da12707ddff65519319c5c93322e"
server_key_iv: "c2cde241f17a43ecc789d2307b28e4a0"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.841302 17369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "3caba0e2eb3c47119e2813d66420e835"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "0a74da12707ddff65519319c5c93322e"
server_key_iv: "c2cde241f17a43ecc789d2307b28e4a0"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.842729 17369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:43:17.843595 17384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.843804 17369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:17.843910 17369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "3caba0e2eb3c47119e2813d66420e835"
format_stamp: "Formatted at 2026-08-09 16:43:17 on dist-test-slave-wl2h"
server_key: "0a74da12707ddff65519319c5c93322e"
server_key_iv: "c2cde241f17a43ecc789d2307b28e4a0"
server_key_version: "encryptionkey@0"
I20260809 16:43:17.843993 17369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:17.876051 17369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:17.876448 17369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:17.876631 17369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:17.876915 17369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:17.877311 17369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:17.877377 17369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.877411 17369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:17.877439 17369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:17.883533 17369 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:34007
I20260809 16:43:17.883584 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:34007 every 8 connection(s)
I20260809 16:43:17.883973 17369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:17.889510 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.884795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:45673 (local address 127.28.62.190:32793)
0809 16:43:17.884862 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.884866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.885087 (+ 221us) server_negotiation.cc:407] Connection header received
0809 16:43:17.885572 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.885573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.885583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.885646 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.886316 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.886812 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.887837 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.888092 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.888146 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.889196 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.889197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.889197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:17.889204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.889219 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:17.889224 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.889286 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.889400 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.889436 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:43:17.889565 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17369
I20260809 16:43:17.889659 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:17.889763 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.884749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:32793 (local address 127.28.62.131:45673)
0809 16:43:17.884952 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.884994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.885488 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.885694 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.885697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.885705 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:17.886195 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.886199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.886901 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.886902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.887755 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.887758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.888166 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.888167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.888357 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.888960 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:17.888961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:17.888990 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:17.889012 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:17.889287 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:17.889292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:17.889295 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.889398 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.889519 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260809 16:43:17.889889 28922 external_mini_cluster.cc:1428] Setting key 205ef0385a57f5dc7f331bb676b91804
I20260809 16:43:17.890555 17524 heartbeater.cc:347] Connected to a master server at 127.28.62.190:32793
I20260809 16:43:17.890676 17524 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:17.890887 17524 heartbeater.cc:511] Master 127.28.62.190:32793 requested a full tablet report, sending...
I20260809 16:43:17.891299 16999 ts_manager.cc:194] Registered new tserver with Master: 3caba0e2eb3c47119e2813d66420e835 (127.28.62.131:34007)
I20260809 16:43:17.891681 16999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.131:45673
I20260809 16:43:17.901933 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:17.906700 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.903571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.28.62.190:32793)
0809 16:43:17.903645 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.903647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.903784 (+ 137us) server_negotiation.cc:407] Connection header received
0809 16:43:17.903853 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.903854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.903868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.903939 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:17.904161 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.904666 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.905451 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.905649 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.905732 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.906426 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:17.906427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:17.906427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:17.906432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:17.906454 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:17.906458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:17.906526 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.906583 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.906608 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:17.908481 16999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:17.910202 16999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:17.916678 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43859 (local address 127.0.0.1:55240)
0809 16:43:17.912875 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.912919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.912996 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.913375 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.913377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.913378 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.913423 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.913425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914307 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.914308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.915553 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.915556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916382 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.916458 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916471 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.916547 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.916601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":64,"threads_started":1}
I20260809 16:43:17.916692 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.28.62.129:43859)
0809 16:43:17.913161 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.913166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.913176 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:17.913217 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.913218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.913235 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.913308 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.913483 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914204 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.915651 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916289 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916383 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916482 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.916563 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.916601 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":92,"threads_started":1}
I20260809 16:43:17.916996 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:34007 (local address 127.0.0.1:49980)
0809 16:43:17.912967 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.912976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.913031 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.913270 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.913273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.913275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.913341 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.913344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914682 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.914683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.915730 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.915732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916750 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.916814 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916818 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.916868 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.916909 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":74,"threads_started":1}
I20260809 16:43:17.917194 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.28.62.130:37173)
0809 16:43:17.913362 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.913366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.913375 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:17.913418 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.913420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.913434 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.913505 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.913699 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914756 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.915973 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916678 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916742 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916816 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.917043 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.917087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":112,"threads_started":1}
I20260809 16:43:17.917326 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.28.62.131:34007)
0809 16:43:17.912842 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:17.912847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:17.913088 (+ 241us) server_negotiation.cc:407] Connection header received
0809 16:43:17.913138 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:17.913141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:17.913167 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:17.913423 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:17.913501 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914587 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.915817 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916550 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916624 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916710 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:17.917194 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 16:43:17.917242 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":139,"threads_started":1}
I20260809 16:43:17.917492 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:17.912745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:37173 (local address 127.0.0.1:50986)
0809 16:43:17.913098 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:17.913107 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:17.913144 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:17.913576 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:17.913578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:17.913580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:17.913620 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.913621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.914839 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.914840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.915881 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:17.915883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:17.916849 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:17.916850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:17.916903 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:17.916906 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:17.916950 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 16:43:17.917001 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":76,"threads_started":1}
I20260809 16:43:17.917644 17135 tablet_service.cc:1467] Processing CreateTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 (DEFAULT_TABLE table=test-table [id=5895bf771fb2432881a13b07235510fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.918032 17135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42524196ff1b495599f1c0f63b1b7bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.919924 17441 tablet_service.cc:1467] Processing CreateTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 (DEFAULT_TABLE table=test-table [id=5895bf771fb2432881a13b07235510fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.920044 17291 tablet_service.cc:1467] Processing CreateTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 (DEFAULT_TABLE table=test-table [id=5895bf771fb2432881a13b07235510fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:17.920270 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42524196ff1b495599f1c0f63b1b7bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.920348 17291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42524196ff1b495599f1c0f63b1b7bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:17.921478 17540 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: Bootstrap starting.
I20260809 16:43:17.922253 17540 tablet_bootstrap.cc:654] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.922643 17540 log.cc:824] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.923427 17542 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Bootstrap starting.
I20260809 16:43:17.923626 17540 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: No bootstrap required, opened a new log
I20260809 16:43:17.923707 17540 ts_tablet_manager.cc:1397] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:17.923960 17543 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Bootstrap starting.
I20260809 16:43:17.924129 17542 tablet_bootstrap.cc:654] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.924460 17542 log.cc:824] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.925076 17543 tablet_bootstrap.cc:654] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:17.925334 17542 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: No bootstrap required, opened a new log
I20260809 16:43:17.925413 17542 ts_tablet_manager.cc:1397] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:17.925426 17543 log.cc:824] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:17.926239 17543 tablet_bootstrap.cc:492] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: No bootstrap required, opened a new log
I20260809 16:43:17.926321 17543 ts_tablet_manager.cc:1397] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:17.926812 17540 raft_consensus.cc:348] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.927098 17540 raft_consensus.cc:374] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.927187 17540 raft_consensus.cc:729] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57282f902cc64fb38dba8778076681c5, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.927341 17540 consensus_queue.cc:260] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.927635 17540 ts_tablet_manager.cc:1428] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 16:43:17.927657 17220 heartbeater.cc:503] Master 127.28.62.190:32793 was elected leader, sending a full tablet report...
I20260809 16:43:17.927784 17542 raft_consensus.cc:348] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.927917 17542 raft_consensus.cc:374] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.927949 17542 raft_consensus.cc:729] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3caba0e2eb3c47119e2813d66420e835, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.928071 17542 consensus_queue.cc:260] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.928326 17542 ts_tablet_manager.cc:1428] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:17.928455 17524 heartbeater.cc:503] Master 127.28.62.190:32793 was elected leader, sending a full tablet report...
I20260809 16:43:17.929478 17543 raft_consensus.cc:348] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.929621 17543 raft_consensus.cc:374] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:17.929644 17543 raft_consensus.cc:729] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2198d4cca03d4c2b88ae6ef821a6eeac, State: Initialized, Role: FOLLOWER
I20260809 16:43:17.929792 17543 consensus_queue.cc:260] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:17.930071 17543 ts_tablet_manager.cc:1428] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 16:43:17.930104 17365 heartbeater.cc:503] Master 127.28.62.190:32793 was elected leader, sending a full tablet report...
W20260809 16:43:17.988013 17367 tablet.cc:2369] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:18.030216 17547 raft_consensus.cc:482] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:18.030423 17547 raft_consensus.cc:504] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:18.030867 17547 leader_election.cc:288] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57282f902cc64fb38dba8778076681c5 (127.28.62.129:43859), 2198d4cca03d4c2b88ae6ef821a6eeac (127.28.62.130:37173)
I20260809 16:43:18.034900 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.031174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:37173 (local address 127.28.62.131:57979)
0809 16:43:18.031240 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:18.031253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:18.031302 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:18.031585 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:18.031586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:18.031587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:18.031699 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:18.031701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.032957 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.032958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:18.033968 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:18.033971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.034638 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:18.034720 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:18.034727 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 16:43:18.034772 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 16:43:18.034807 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":23}
I20260809 16:43:18.034937 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.031275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:57979 (local address 127.28.62.130:37173)
0809 16:43:18.031352 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:18.031354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:18.031364 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:18.031405 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:18.031406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:18.031416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:18.031538 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:18.031828 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.032718 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034074 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.034531 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:18.034611 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:18.034808 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 16:43:18.034858 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 16:43:18.035056 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.031275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:55073 (local address 127.28.62.129:43859)
0809 16:43:18.031334 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:18.031341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:18.031461 (+ 120us) server_negotiation.cc:407] Connection header received
0809 16:43:18.031494 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:18.031495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:18.031508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:18.031573 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:18.031833 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.032720 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034173 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.034557 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034666 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:18.034675 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:18.034901 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 16:43:18.034953 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:18.035050 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.031194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43859 (local address 127.28.62.131:55073)
0809 16:43:18.031378 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:18.031389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:18.031440 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:18.031589 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:18.031590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:18.031591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:18.031699 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:18.031701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.032868 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.032869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:18.034071 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:18.034075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.034698 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.034699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:18.034814 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:18.034818 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:18.034880 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 16:43:18.034923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":63,"thread_start_us":106,"threads_started":1}
I20260809 16:43:18.035485 17167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42524196ff1b495599f1c0f63b1b7bf1" candidate_uuid: "3caba0e2eb3c47119e2813d66420e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57282f902cc64fb38dba8778076681c5" is_pre_election: true
I20260809 16:43:18.035684 17167 raft_consensus.cc:2454] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3caba0e2eb3c47119e2813d66420e835 in term 0.
I20260809 16:43:18.035687 17315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42524196ff1b495599f1c0f63b1b7bf1" candidate_uuid: "3caba0e2eb3c47119e2813d66420e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" is_pre_election: true
I20260809 16:43:18.035890 17315 raft_consensus.cc:2454] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3caba0e2eb3c47119e2813d66420e835 in term 0.
I20260809 16:43:18.035980 17385 leader_election.cc:302] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3caba0e2eb3c47119e2813d66420e835, 57282f902cc64fb38dba8778076681c5; no voters:
I20260809 16:43:18.036131 17547 raft_consensus.cc:2781] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:18.036224 17547 raft_consensus.cc:482] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:18.036263 17547 raft_consensus.cc:3037] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:18.037129 17547 raft_consensus.cc:504] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:18.037297 17547 leader_election.cc:288] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [CANDIDATE]: Term 1 election: Requested vote from peers 57282f902cc64fb38dba8778076681c5 (127.28.62.129:43859), 2198d4cca03d4c2b88ae6ef821a6eeac (127.28.62.130:37173)
I20260809 16:43:18.037470 17315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42524196ff1b495599f1c0f63b1b7bf1" candidate_uuid: "3caba0e2eb3c47119e2813d66420e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac"
I20260809 16:43:18.037503 17167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42524196ff1b495599f1c0f63b1b7bf1" candidate_uuid: "3caba0e2eb3c47119e2813d66420e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57282f902cc64fb38dba8778076681c5"
I20260809 16:43:18.037550 17315 raft_consensus.cc:3037] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:18.037631 17167 raft_consensus.cc:3037] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:18.038455 17315 raft_consensus.cc:2454] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3caba0e2eb3c47119e2813d66420e835 in term 1.
I20260809 16:43:18.038712 17386 leader_election.cc:302] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2198d4cca03d4c2b88ae6ef821a6eeac, 3caba0e2eb3c47119e2813d66420e835; no voters:
I20260809 16:43:18.038790 17167 raft_consensus.cc:2454] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3caba0e2eb3c47119e2813d66420e835 in term 1.
I20260809 16:43:18.038825 17547 raft_consensus.cc:2781] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:18.038991 17547 raft_consensus.cc:686] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 1 LEADER]: Becoming Leader. State: Replica: 3caba0e2eb3c47119e2813d66420e835, State: Running, Role: LEADER
I20260809 16:43:18.039139 17547 consensus_queue.cc:237] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } }
I20260809 16:43:18.040114 16998 catalog_manager.cc:5495] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3caba0e2eb3c47119e2813d66420e835 (127.28.62.131). New cstate: current_term: 1 leader_uuid: "3caba0e2eb3c47119e2813d66420e835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3caba0e2eb3c47119e2813d66420e835" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 } health_report { overall_health: UNKNOWN } } }
I20260809 16:43:18.067219 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.064570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.28.62.131:34007)
0809 16:43:18.064653 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:18.064655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:18.064666 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:43:18.064711 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:18.064712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:18.064736 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:18.064852 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:18.065000 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.065497 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.066288 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:18.066514 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:18.066601 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:18.066772 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:18.066893 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:18.066986 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:18.067083 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 16:43:18.067105 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 16:43:18.069238 17315 raft_consensus.cc:1261] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 1 FOLLOWER]: Refusing update from remote peer 3caba0e2eb3c47119e2813d66420e835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:18.069319 17167 raft_consensus.cc:1261] T 42524196ff1b495599f1c0f63b1b7bf1 P 57282f902cc64fb38dba8778076681c5 [term 1 FOLLOWER]: Refusing update from remote peer 3caba0e2eb3c47119e2813d66420e835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:18.069665 17550 consensus_queue.cc:1035] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2198d4cca03d4c2b88ae6ef821a6eeac" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:18.069813 17547 consensus_queue.cc:1035] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57282f902cc64fb38dba8778076681c5" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:18.074739 16999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:18.074843 16999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:18.075692 16999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e3cd0cb2b614c6ab409d0bd17174dac: Sending DeleteTablet for 3 replicas of tablet 42524196ff1b495599f1c0f63b1b7bf1
I20260809 16:43:18.075956 17441 tablet_service.cc:1514] Processing DeleteTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:18 UTC) from {username='slave'} at 127.0.0.1:49980
I20260809 16:43:18.076076 17291 tablet_service.cc:1514] Processing DeleteTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:18 UTC) from {username='slave'} at 127.0.0.1:50986
I20260809 16:43:18.076089 17135 tablet_service.cc:1514] Processing DeleteTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:18 UTC) from {username='slave'} at 127.0.0.1:55240
I20260809 16:43:18.076274 17290 tablet_service.cc:1514] Processing DeleteTablet for tablet 42524196ff1b495599f1c0f63b1b7bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:18 UTC) from {username='slave'} at 127.0.0.1:50986
I20260809 16:43:18.076377 17565 tablet_replica.cc:330] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: stopping tablet replica
I20260809 16:43:18.076411 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17068
I20260809 16:43:18.076481 17565 raft_consensus.cc:2229] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:18.076570 17565 raft_consensus.cc:2258] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:18.081370 17565 ts_tablet_manager.cc:1905] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:18.081954 17566 tablet_replica.cc:330] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: stopping tablet replica
I20260809 16:43:18.082093 17566 raft_consensus.cc:2229] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:18.082237 17566 raft_consensus.cc:2258] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:18.082834 17566 ts_tablet_manager.cc:1905] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:18.083839 17565 ts_tablet_manager.cc:1918] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:18.083946 17565 log.cc:1196] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/42524196ff1b495599f1c0f63b1b7bf1
I20260809 16:43:18.084299 17566 ts_tablet_manager.cc:1918] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:18.084306 17565 ts_tablet_manager.cc:1939] T 42524196ff1b495599f1c0f63b1b7bf1 P 3caba0e2eb3c47119e2813d66420e835: Deleting consensus metadata
I20260809 16:43:18.084398 17566 log.cc:1196] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/42524196ff1b495599f1c0f63b1b7bf1
I20260809 16:43:18.084622 17566 ts_tablet_manager.cc:1939] T 42524196ff1b495599f1c0f63b1b7bf1 P 2198d4cca03d4c2b88ae6ef821a6eeac: Deleting consensus metadata
I20260809 16:43:18.087430 16977 catalog_manager.cc:4841] TS 2198d4cca03d4c2b88ae6ef821a6eeac (127.28.62.130:37173): tablet 42524196ff1b495599f1c0f63b1b7bf1 (table test-table [id=5895bf771fb2432881a13b07235510fa]) successfully deleted
W20260809 16:43:18.087625 16977 catalog_manager.cc:4808] TS 2198d4cca03d4c2b88ae6ef821a6eeac (127.28.62.130:37173): delete failed for tablet 42524196ff1b495599f1c0f63b1b7bf1 because the tablet was not found. No further retry: Not found: Tablet not found: 42524196ff1b495599f1c0f63b1b7bf1
I20260809 16:43:18.087721 16977 catalog_manager.cc:4841] TS 3caba0e2eb3c47119e2813d66420e835 (127.28.62.131:34007): tablet 42524196ff1b495599f1c0f63b1b7bf1 (table test-table [id=5895bf771fb2432881a13b07235510fa]) successfully deleted
W20260809 16:43:18.092305 16976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:43859 (error 108)
I20260809 16:43:18.092906 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:18.092731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:43859 (local address 127.0.0.1:55244)
0809 16:43:18.092792 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:18.092843 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:43859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 16:43:18.092979 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17223
W20260809 16:43:18.095963 16976 catalog_manager.cc:4584] TS 57282f902cc64fb38dba8778076681c5 (127.28.62.129:43859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42524196ff1b495599f1c0f63b1b7bf1: Network error: Client connection negotiation failed: client connection to 127.28.62.129:43859: connect: Connection refused (error 111)
I20260809 16:43:18.100481 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17369
I20260809 16:43:18.109277 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 16955
2026-08-09T16:43:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:18 dist-test-slave-wl2h krb5kdc[17571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:18 dist-test-slave-wl2h krb5kdc[17571](info): set up 2 sockets
Aug 09 16:43:18 dist-test-slave-wl2h krb5kdc[17571](info): commencing operation
krb5kdc: starting...
W20260809 16:43:19.249300 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:19.378650 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45247
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:45247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:19.463485 17587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:19.463743 17587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:19.463783 17587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:19.464919 17587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:19.464993 17587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:19.465021 17587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:19.465039 17587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:19.465060 17587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:19.466566 17587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:45247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:19.466861 17587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:19.467175 17587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:19.472208 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:19.472198 17593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:19.472280 17587 server_base.cc:1034] running on GCE node
W20260809 16:43:19.472198 17594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:19.472766 17587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:19.473016 17587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:19.474169 17587 hybrid_clock.cc:648] HybridClock initialized: now 1786293799474144 us; error 43 us; skew 500 ppm
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:19.476907 17587 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:19.477300 17587 webserver.cc:460] Webserver started at http://127.28.62.190:34567/ using document root <none> and password file <none>
I20260809 16:43:19.477499 17587 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:19.477541 17587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:19.477640 17587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:19.478667 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "7e8c4486dcea4e90a9de2f67e518ea8d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "172cfffff2ca348f9ade3bef28749c3e"
server_key_iv: "0e2dc37d8e8da834085f1732c2645af7"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.478963 17587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "7e8c4486dcea4e90a9de2f67e518ea8d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "172cfffff2ca348f9ade3bef28749c3e"
server_key_iv: "0e2dc37d8e8da834085f1732c2645af7"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.480358 17587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:19.481122 17606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.481307 17587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260809 16:43:19.481391 17587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "7e8c4486dcea4e90a9de2f67e518ea8d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "172cfffff2ca348f9ade3bef28749c3e"
server_key_iv: "0e2dc37d8e8da834085f1732c2645af7"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.481448 17587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:19.498312 17587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:19.500801 17587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:19.500929 17587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:19.505100 17587 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:45247
I20260809 16:43:19.505162 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:45247 every 8 connection(s)
I20260809 16:43:19.505537 17587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:19.505964 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17587
I20260809 16:43:19.506145 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:19.506392 28922 external_mini_cluster.cc:1428] Setting key 3d06d5d5d8e01ea5b0f411c5025eb614
I20260809 16:43:19.506800 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:19.510517 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: Bootstrap starting.
I20260809 16:43:19.511317 17668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:19.511745 17668 log.cc:824] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:19.512573 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: No bootstrap required, opened a new log
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:19.514951 17668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } }
I20260809 16:43:19.515182 17668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:19.515216 17668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8c4486dcea4e90a9de2f67e518ea8d, State: Initialized, Role: FOLLOWER
I20260809 16:43:19.515386 17668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } }
I20260809 16:43:19.515484 17668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:19.515534 17668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:19.515604 17668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:19.516222 17668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } }
I20260809 16:43:19.516360 17668 leader_election.cc:302] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e8c4486dcea4e90a9de2f67e518ea8d; no voters:
I20260809 16:43:19.516615 17668 leader_election.cc:288] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:19.516649 17674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:19.516789 17674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [term 1 LEADER]: Becoming Leader. State: Replica: 7e8c4486dcea4e90a9de2f67e518ea8d, State: Running, Role: LEADER
I20260809 16:43:19.516880 17674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } }
I20260809 16:43:19.516914 17668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:19.517244 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8c4486dcea4e90a9de2f67e518ea8d. Latest consensus state: current_term: 1 leader_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } } }
I20260809 16:43:19.517254 17676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8c4486dcea4e90a9de2f67e518ea8d" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45247 } } }
I20260809 16:43:19.517338 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:19.517376 17676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:19.517899 17680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:19.518422 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:19.506982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.28.62.190:45247)
0809 16:43:19.507203 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:19.507210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:19.507236 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:43:19.507653 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:19.507659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:19.507703 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:19.507815 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:19.508782 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.509644 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.510816 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.511286 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.511376 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:19.514221 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:19.514225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:19.514225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:19.514235 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:19.516124 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.516556 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.516559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.516559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.516651 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.516918 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.516919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.516919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.517287 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:19.517511 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:19.518044 (+ 533us) server_negotiation.cc:299] Negotiation successful
0809 16:43:19.518195 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260809 16:43:19.518589 17680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:19.520249 17680 catalog_manager.cc:1347] Generated new cluster ID: 6a1c045e2713441fab9ff87e6c827b9f
I20260809 16:43:19.520303 17680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:19.543569 17680 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:19.544164 17680 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:19.550356 17680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: Generated new TSK 0
I20260809 16:43:19.550571 17680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:43:19.589638 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:19.690590 17698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:19.690821 17698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:19.690845 17698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:19.691946 17698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:19.691987 17698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:19.692013 17698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:19.693425 17698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:19.693638 17698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:19.693818 17698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:19.699434 17706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:19.699442 17704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:19.699440 17703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:19.699683 17698 server_base.cc:1034] running on GCE node
I20260809 16:43:19.699934 17698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:19.700182 17698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:19.701351 17698 hybrid_clock.cc:648] HybridClock initialized: now 1786293799701337 us; error 32 us; skew 500 ppm
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:19.704304 17698 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:43:19.704775 17698 webserver.cc:460] Webserver started at http://127.28.62.129:46651/ using document root <none> and password file <none>
I20260809 16:43:19.705032 17698 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:19.705088 17698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:19.705224 17698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:19.706315 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "1ac3e1cf62a04278a3ac740f3e7d3423"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "2d89a56c2f64a9140b6e68fb6a11aa1c"
server_key_iv: "7143573ee8c244e2da73b79dc5fe3231"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.706686 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac3e1cf62a04278a3ac740f3e7d3423"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "2d89a56c2f64a9140b6e68fb6a11aa1c"
server_key_iv: "7143573ee8c244e2da73b79dc5fe3231"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.708268 17698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:19.709098 17715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.709287 17698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:19.709401 17698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "1ac3e1cf62a04278a3ac740f3e7d3423"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "2d89a56c2f64a9140b6e68fb6a11aa1c"
server_key_iv: "7143573ee8c244e2da73b79dc5fe3231"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.709481 17698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:19.737425 17698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:19.740120 17698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:19.740317 17698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:19.740602 17698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:19.740948 17698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:19.741014 17698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.741072 17698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:19.741108 17698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.748018 17698 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:34273
I20260809 16:43:19.748100 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:34273 every 8 connection(s)
I20260809 16:43:19.748481 17698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:19.757027 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17698
I20260809 16:43:19.757220 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:19.757479 28922 external_mini_cluster.cc:1428] Setting key 07a38f46054e833e214442d1403b8036
I20260809 16:43:19.758548 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:19.749339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:47571 (local address 127.28.62.190:45247)
0809 16:43:19.749388 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:19.749390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:19.749595 (+ 205us) server_negotiation.cc:407] Connection header received
0809 16:43:19.750149 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:19.750150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:19.750160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:19.750226 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:19.751307 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.751867 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.752899 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.753133 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.753199 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:19.755399 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:19.755400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:19.755401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:19.755415 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:19.756826 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.757411 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.757412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.757412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.757469 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.757811 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.757812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.757813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.757972 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:19.758044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:19.758395 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 16:43:19.758484 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:19.758880 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:19.749274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45247 (local address 127.28.62.129:47571)
0809 16:43:19.749457 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:19.749502 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:19.749994 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:19.750310 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:19.750314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:19.750897 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:19.751217 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:19.751223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.751956 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.751957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:19.752796 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:19.752799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.753227 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.753228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:19.753341 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:19.753868 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:19.753876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:19.755304 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:19.756930 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:19.756940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:19.756942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:19.757322 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:19.757550 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:19.757551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:19.757551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:19.757736 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:19.758045 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:19.758052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:19.758232 (+ 180us) client_negotiation.cc:769] Sending connection context
0809 16:43:19.758345 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 16:43:19.758571 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260809 16:43:19.759485 17854 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45247
I20260809 16:43:19.759608 17854 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:19.759824 17854 heartbeater.cc:511] Master 127.28.62.190:45247 requested a full tablet report, sending...
I20260809 16:43:19.760308 17624 ts_manager.cc:194] Registered new tserver with Master: 1ac3e1cf62a04278a3ac740f3e7d3423 (127.28.62.129:34273)
I20260809 16:43:19.761291 17624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:47571
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:43:19.787595 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:19.871676 17861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:19.871883 17861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:19.871907 17861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:19.873131 17861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:19.873178 17861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:19.873206 17861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:19.874740 17861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:19.875030 17861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:19.875319 17861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 16:43:19.880915 17861 server_base.cc:1034] running on GCE node
W20260809 16:43:19.880834 17868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:19.880859 17871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:19.880820 17869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:19.881283 17861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:19.881587 17861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:19.882778 17861 hybrid_clock.cc:648] HybridClock initialized: now 1786293799882759 us; error 41 us; skew 500 ppm
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:19.885582 17861 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:43:19.886137 17861 webserver.cc:460] Webserver started at http://127.28.62.130:44615/ using document root <none> and password file <none>
I20260809 16:43:19.886384 17861 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:19.886456 17861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:19.886620 17861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:19.887840 17861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "d49190ef2dc14476b3fc870c28cbb72d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "bad6fc63c4e166b2ccba052ba4c7af03"
server_key_iv: "0c73d8c96f327b14a24341a2cdc383d6"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.888229 17861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "d49190ef2dc14476b3fc870c28cbb72d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "bad6fc63c4e166b2ccba052ba4c7af03"
server_key_iv: "0c73d8c96f327b14a24341a2cdc383d6"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.889741 17861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 16:43:19.890610 17879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.890841 17861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:19.890959 17861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "d49190ef2dc14476b3fc870c28cbb72d"
format_stamp: "Formatted at 2026-08-09 16:43:19 on dist-test-slave-wl2h"
server_key: "bad6fc63c4e166b2ccba052ba4c7af03"
server_key_iv: "0c73d8c96f327b14a24341a2cdc383d6"
server_key_version: "encryptionkey@0"
I20260809 16:43:19.891067 17861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:19.909412 17861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:19.911688 17861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:19.911860 17861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:19.912142 17861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:19.912549 17861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:19.912614 17861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.912662 17861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:19.912714 17861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:19.919394 17861 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:37333
I20260809 16:43:19.919446 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:37333 every 8 connection(s)
I20260809 16:43:19.919780 17861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:19.923863 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 17861
I20260809 16:43:19.923985 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:19.924248 28922 external_mini_cluster.cc:1428] Setting key 90fcd649eecb4c98e6902f018eed8529
Aug 09 16:43:19 dist-test-slave-wl2h krb5kdc[17571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:19.930362 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:19.920547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:59421 (local address 127.28.62.190:45247)
0809 16:43:19.920596 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:19.920598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:19.920833 (+ 235us) server_negotiation.cc:407] Connection header received
0809 16:43:19.921302 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:19.921303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:19.921316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:19.921381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:19.922373 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.922901 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.924191 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.924458 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.924558 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:19.926825 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:19.926826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:19.926827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:19.926834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:19.928662 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.929186 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.929188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.929188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.929258 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:19.929543 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:19.929544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:19.929545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:19.929698 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:19.929781 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:19.930163 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 16:43:19.930258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 16:43:19.930462 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:19.920490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45247 (local address 127.28.62.130:59421)
0809 16:43:19.920674 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:19.920718 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:19.921209 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:19.921451 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:19.921455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:19.922030 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:19.922274 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:19.922279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.923121 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.923122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:19.924077 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:19.924088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:19.924562 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:19.924563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:19.924715 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:19.925313 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:19.925328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:19.926720 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:19.928764 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:19.928772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:19.928775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:19.929049 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:19.929341 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:19.929342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:19.929342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:19.929462 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:19.929815 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:19.929824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:19.929989 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 16:43:19.930111 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 16:43:19.930250 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260809 16:43:19.931123 18009 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45247
I20260809 16:43:19.931227 18009 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:19.931425 18009 heartbeater.cc:511] Master 127.28.62.190:45247 requested a full tablet report, sending...
I20260809 16:43:19.931936 17624 ts_manager.cc:194] Registered new tserver with Master: d49190ef2dc14476b3fc870c28cbb72d (127.28.62.130:37333)
I20260809 16:43:19.932374 17624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:59421
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:43:19.955169 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:20.043936 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:20.044152 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:20.044176 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:20.045332 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:20.045378 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:20.045404 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:20.046860 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:20.047147 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:20.047417 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:20.052806 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.052815 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.052816 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.053264 18015 server_base.cc:1034] running on GCE node
I20260809 16:43:20.053460 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:20.053656 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:20.054795 18015 hybrid_clock.cc:648] HybridClock initialized: now 1786293800054784 us; error 28 us; skew 500 ppm
Aug 09 16:43:20 dist-test-slave-wl2h krb5kdc[17571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293800, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:20.057480 18015 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:43:20.057950 18015 webserver.cc:460] Webserver started at http://127.28.62.131:37067/ using document root <none> and password file <none>
I20260809 16:43:20.058171 18015 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:20.058220 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:20.058313 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:20.059396 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "41cd206f2fa24f258261ccc887cbbda7"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "d5d0b55c725cfd2bc49f1f6a33031bab"
server_key_iv: "4a116f82a15d38bf3e753fdeafb74212"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.059705 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "41cd206f2fa24f258261ccc887cbbda7"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "d5d0b55c725cfd2bc49f1f6a33031bab"
server_key_iv: "4a116f82a15d38bf3e753fdeafb74212"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.061452 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:20.062204 18034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.062392 18015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:20.062497 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "41cd206f2fa24f258261ccc887cbbda7"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "d5d0b55c725cfd2bc49f1f6a33031bab"
server_key_iv: "4a116f82a15d38bf3e753fdeafb74212"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.062608 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:20.088507 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:20.090490 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:20.090677 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:20.090986 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:20.091367 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:20.091430 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.091475 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:20.091503 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.098084 18015 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:37447
I20260809 16:43:20.098138 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:37447 every 8 connection(s)
I20260809 16:43:20.098501 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:20.101212 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18015
I20260809 16:43:20.101325 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:20.101497 28922 external_mini_cluster.cc:1428] Setting key fffa9f765876d701eeb5354019293181
Aug 09 16:43:20 dist-test-slave-wl2h krb5kdc[17571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293800, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:20.108201 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.099337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:42403 (local address 127.28.62.190:45247)
0809 16:43:20.099402 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.099404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.099614 (+ 210us) server_negotiation.cc:407] Connection header received
0809 16:43:20.100071 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.100072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.100083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.100156 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.101151 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.101888 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.102938 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.103225 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.103313 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.105383 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.105385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.105385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:20.105393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.106683 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:20.107197 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:20.107198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:20.107199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:20.107263 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:20.107529 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:20.107530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:20.107531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:20.107662 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.107740 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.108019 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.108109 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:43:20.108394 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.099288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45247 (local address 127.28.62.131:42403)
0809 16:43:20.099453 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.099495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.099987 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.100263 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.100266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.100826 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:20.101017 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.101021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.102000 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.102001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.102819 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.102821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.103327 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.103328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.103468 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.103914 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:20.103921 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:20.105268 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:20.106806 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:20.106813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:20.106815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:20.107112 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:20.107346 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:20.107346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:20.107347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:20.107458 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:20.107742 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:20.107749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:20.107883 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.107995 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.108162 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260809 16:43:20.108883 18166 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45247
I20260809 16:43:20.108960 18166 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:20.109159 18166 heartbeater.cc:511] Master 127.28.62.190:45247 requested a full tablet report, sending...
I20260809 16:43:20.109572 17624 ts_manager.cc:194] Registered new tserver with Master: 41cd206f2fa24f258261ccc887cbbda7 (127.28.62.131:37447)
I20260809 16:43:20.109958 17624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:42403
I20260809 16:43:20.113253 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:20.121366 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.114806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.28.62.190:45247)
0809 16:43:20.114884 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.114886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.114967 (+ 81us) server_negotiation.cc:407] Connection header received
0809 16:43:20.115134 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.115135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.115146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.115204 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.115758 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.116225 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.116986 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.117241 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.117318 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.118734 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.118735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.118736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:20.118743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.120092 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:20.120498 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:20.120499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:20.120499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:20.120555 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:20.120790 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:20.120791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:20.120792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:20.120931 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.121052 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.121204 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.121269 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 16:43:20.123111 17624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:20.124933 17624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:20.132184 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.127867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.28.62.131:37447)
0809 16:43:20.128062 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.128067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.128080 (+ 13us) server_negotiation.cc:407] Connection header received
0809 16:43:20.128120 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.128123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.128140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.128236 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.128482 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129617 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.130915 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.131690 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131771 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.131834 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.132039 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.132099 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260809 16:43:20.132203 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.127624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:37447 (local address 127.0.0.1:50850)
0809 16:43:20.127884 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.127928 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.128009 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.128287 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.128290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.128293 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.128381 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.128382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129747 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.129748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.130811 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.130814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.131791 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.131907 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.131924 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.132010 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.132133 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":31,"thread_start_us":94,"threads_started":1}
I20260809 16:43:20.132373 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.128060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.28.62.129:34273)
0809 16:43:20.128389 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.128393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.128404 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:43:20.128544 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.128546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.128565 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.128693 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.128811 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129848 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131116 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.131819 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131891 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.131928 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.132249 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.132300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260809 16:43:20.132202 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.127821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34273 (local address 127.0.0.1:54288)
0809 16:43:20.128351 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.128361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.128406 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.128687 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.128687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.128697 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.128748 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.128749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129956 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.129957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.131049 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.131051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.131927 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.131989 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.131994 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.132067 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.132110 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":79,"threads_started":1}
I20260809 16:43:20.133183 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.127624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:37333 (local address 127.0.0.1:44406)
0809 16:43:20.127976 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.127985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.128027 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.128739 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.128739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.128741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.128772 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.128773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129706 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.129707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.130971 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.130973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.132954 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.132954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.133015 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.133019 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.133076 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.133117 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260809 16:43:20.133215 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.127781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.28.62.130:37333)
0809 16:43:20.128549 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.128554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.128565 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:43:20.128621 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.128624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.128641 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.128844 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.128923 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.129595 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.131957 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.132848 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.132950 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.133009 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.133089 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.133133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":103,"threads_started":1}
I20260809 16:43:20.133347 17775 tablet_service.cc:1467] Processing CreateTablet for tablet b301dbd97165489aae0804e1d349c6b3 (DEFAULT_TABLE table=test-table [id=5605904091d347e3a2d28ff2db75ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.133467 18090 tablet_service.cc:1467] Processing CreateTablet for tablet b301dbd97165489aae0804e1d349c6b3 (DEFAULT_TABLE table=test-table [id=5605904091d347e3a2d28ff2db75ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.133755 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b301dbd97165489aae0804e1d349c6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.133754 17775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b301dbd97165489aae0804e1d349c6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.133869 17933 tablet_service.cc:1467] Processing CreateTablet for tablet b301dbd97165489aae0804e1d349c6b3 (DEFAULT_TABLE table=test-table [id=5605904091d347e3a2d28ff2db75ea1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.134160 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b301dbd97165489aae0804e1d349c6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.136343 18184 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Bootstrap starting.
I20260809 16:43:20.136950 18184 tablet_bootstrap.cc:654] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.137180 18185 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: Bootstrap starting.
I20260809 16:43:20.137679 18184 log.cc:824] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.138216 18185 tablet_bootstrap.cc:654] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.138710 18184 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: No bootstrap required, opened a new log
I20260809 16:43:20.138792 18184 ts_tablet_manager.cc:1397] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:20.138805 18185 log.cc:824] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.139601 18185 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: No bootstrap required, opened a new log
I20260809 16:43:20.139679 18185 ts_tablet_manager.cc:1397] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 16:43:20.140777 18184 raft_consensus.cc:348] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.140965 18184 raft_consensus.cc:374] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.141005 18184 raft_consensus.cc:729] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41cd206f2fa24f258261ccc887cbbda7, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.141125 18184 consensus_queue.cc:260] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.141420 18184 ts_tablet_manager.cc:1428] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:20.141456 18166 heartbeater.cc:503] Master 127.28.62.190:45247 was elected leader, sending a full tablet report...
I20260809 16:43:20.141681 18186 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Bootstrap starting.
I20260809 16:43:20.141654 18185 raft_consensus.cc:348] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.141815 18185 raft_consensus.cc:374] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.141860 18185 raft_consensus.cc:729] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac3e1cf62a04278a3ac740f3e7d3423, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.142012 18185 consensus_queue.cc:260] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.142303 18185 ts_tablet_manager.cc:1428] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 16:43:20.142395 18186 tablet_bootstrap.cc:654] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.142452 17854 heartbeater.cc:503] Master 127.28.62.190:45247 was elected leader, sending a full tablet report...
I20260809 16:43:20.143004 18186 log.cc:824] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.143874 18186 tablet_bootstrap.cc:492] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: No bootstrap required, opened a new log
I20260809 16:43:20.143929 18186 ts_tablet_manager.cc:1397] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:20.145260 18186 raft_consensus.cc:348] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.145386 18186 raft_consensus.cc:374] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.145408 18186 raft_consensus.cc:729] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49190ef2dc14476b3fc870c28cbb72d, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.145529 18186 consensus_queue.cc:260] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.145812 18186 ts_tablet_manager.cc:1428] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:20.145892 18009 heartbeater.cc:503] Master 127.28.62.190:45247 was elected leader, sending a full tablet report...
I20260809 16:43:20.152801 18192 raft_consensus.cc:482] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:20.152876 18192 raft_consensus.cc:504] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.153170 18192 leader_election.cc:288] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41cd206f2fa24f258261ccc887cbbda7 (127.28.62.131:37447), 1ac3e1cf62a04278a3ac740f3e7d3423 (127.28.62.129:34273)
I20260809 16:43:20.155916 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.153318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34273 (local address 127.28.62.130:48429)
0809 16:43:20.153368 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.153382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.153428 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.153567 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.153568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.153570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.153643 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.153644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.154300 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.154301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.155088 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.155091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.155656 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.155740 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.155743 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.155811 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.155850 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 16:43:20.155916 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.153391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:48429 (local address 127.28.62.129:34273)
0809 16:43:20.153443 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.153446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.153455 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:20.153484 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.153485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.153496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.153548 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.153718 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.154209 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155180 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.155559 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155649 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.155658 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.155816 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.155854 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 16:43:20.156129 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.153334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:37447 (local address 127.28.62.130:47309)
0809 16:43:20.153508 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.153518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.153585 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.153755 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.153756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.153757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.153835 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.153836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.154465 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.154466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.155280 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.155283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.155848 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.155920 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.155923 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.156002 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.156048 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260809 16:43:20.156299 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b301dbd97165489aae0804e1d349c6b3" candidate_uuid: "d49190ef2dc14476b3fc870c28cbb72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" is_pre_election: true
I20260809 16:43:20.156458 17798 raft_consensus.cc:2454] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49190ef2dc14476b3fc870c28cbb72d in term 0.
I20260809 16:43:20.156500 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.153474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:47309 (local address 127.28.62.131:37447)
0809 16:43:20.153519 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.153521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.153558 (+ 37us) server_negotiation.cc:407] Connection header received
0809 16:43:20.153666 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.153667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.153676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.153753 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.153907 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.154376 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155370 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.155748 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.155847 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.155854 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.156360 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.156401 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:43:20.156709 17883 leader_election.cc:302] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac3e1cf62a04278a3ac740f3e7d3423, d49190ef2dc14476b3fc870c28cbb72d; no voters:
I20260809 16:43:20.156759 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b301dbd97165489aae0804e1d349c6b3" candidate_uuid: "d49190ef2dc14476b3fc870c28cbb72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cd206f2fa24f258261ccc887cbbda7" is_pre_election: true
I20260809 16:43:20.156847 18192 raft_consensus.cc:2781] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:20.156899 18192 raft_consensus.cc:482] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:20.156930 18192 raft_consensus.cc:3037] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.156937 18112 raft_consensus.cc:2454] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49190ef2dc14476b3fc870c28cbb72d in term 0.
I20260809 16:43:20.157652 18192 raft_consensus.cc:504] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.157866 18192 leader_election.cc:288] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [CANDIDATE]: Term 1 election: Requested vote from peers 41cd206f2fa24f258261ccc887cbbda7 (127.28.62.131:37447), 1ac3e1cf62a04278a3ac740f3e7d3423 (127.28.62.129:34273)
I20260809 16:43:20.157930 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b301dbd97165489aae0804e1d349c6b3" candidate_uuid: "d49190ef2dc14476b3fc870c28cbb72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cd206f2fa24f258261ccc887cbbda7"
I20260809 16:43:20.157938 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b301dbd97165489aae0804e1d349c6b3" candidate_uuid: "d49190ef2dc14476b3fc870c28cbb72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423"
I20260809 16:43:20.158012 17798 raft_consensus.cc:3037] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.158012 18112 raft_consensus.cc:3037] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.158972 18112 raft_consensus.cc:2454] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49190ef2dc14476b3fc870c28cbb72d in term 1.
I20260809 16:43:20.158972 17798 raft_consensus.cc:2454] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49190ef2dc14476b3fc870c28cbb72d in term 1.
I20260809 16:43:20.159242 17881 leader_election.cc:302] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41cd206f2fa24f258261ccc887cbbda7, d49190ef2dc14476b3fc870c28cbb72d; no voters:
I20260809 16:43:20.159372 18192 raft_consensus.cc:2781] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:20.159520 18192 raft_consensus.cc:686] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 1 LEADER]: Becoming Leader. State: Replica: d49190ef2dc14476b3fc870c28cbb72d, State: Running, Role: LEADER
I20260809 16:43:20.159634 18192 consensus_queue.cc:237] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } }
I20260809 16:43:20.160472 17624 catalog_manager.cc:5495] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d reported cstate change: term changed from 0 to 1, leader changed from <none> to d49190ef2dc14476b3fc870c28cbb72d (127.28.62.130). New cstate: current_term: 1 leader_uuid: "d49190ef2dc14476b3fc870c28cbb72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49190ef2dc14476b3fc870c28cbb72d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 } health_report { overall_health: UNKNOWN } } }
I20260809 16:43:20.170141 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.167986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.28.62.130:37333)
0809 16:43:20.168042 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.168044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.168052 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:20.168088 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.168088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.168109 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.168182 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:20.168299 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.168752 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.169457 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.169651 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.169706 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.169849 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:20.169928 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:20.169998 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.170079 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.170089 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 16:43:20.170368 18010 tablet.cc:2369] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:20.172088 18112 raft_consensus.cc:1261] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 1 FOLLOWER]: Refusing update from remote peer d49190ef2dc14476b3fc870c28cbb72d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:20.172087 17798 raft_consensus.cc:1261] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 1 FOLLOWER]: Refusing update from remote peer d49190ef2dc14476b3fc870c28cbb72d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:20.172462 18194 consensus_queue.cc:1035] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac3e1cf62a04278a3ac740f3e7d3423" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:20.172608 18192 consensus_queue.cc:1035] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cd206f2fa24f258261ccc887cbbda7" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:20.176159 18199 mvcc.cc:205] Tried to move back new op lower bound from 7316659405501763584 to 7316659405454573568. Current Snapshot: MvccSnapshot[applied={T|T < 7316659405501763584 or (T in {7316659405501763584})}]
I20260809 16:43:20.176486 18200 mvcc.cc:205] Tried to move back new op lower bound from 7316659405501763584 to 7316659405454573568. Current Snapshot: MvccSnapshot[applied={T|T < 7316659405501763584}]
I20260809 16:43:20.177570 17623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:20.177692 17623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:20.178511 17623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e8c4486dcea4e90a9de2f67e518ea8d: Sending DeleteTablet for 3 replicas of tablet b301dbd97165489aae0804e1d349c6b3
I20260809 16:43:20.178839 17933 tablet_service.cc:1514] Processing DeleteTablet for tablet b301dbd97165489aae0804e1d349c6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:44406
I20260809 16:43:20.178890 17775 tablet_service.cc:1514] Processing DeleteTablet for tablet b301dbd97165489aae0804e1d349c6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:54288
I20260809 16:43:20.178890 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet b301dbd97165489aae0804e1d349c6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:50850
I20260809 16:43:20.179155 18208 tablet_replica.cc:330] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423: stopping tablet replica
I20260809 16:43:20.179219 18209 tablet_replica.cc:330] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: stopping tablet replica
I20260809 16:43:20.179261 18208 raft_consensus.cc:2229] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:20.179294 18209 raft_consensus.cc:2229] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:20.179358 18208 raft_consensus.cc:2258] T b301dbd97165489aae0804e1d349c6b3 P 1ac3e1cf62a04278a3ac740f3e7d3423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.179373 18209 raft_consensus.cc:2258] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.179517 18207 tablet_replica.cc:330] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: stopping tablet replica
I20260809 16:43:20.179589 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17698
I20260809 16:43:20.179625 18207 raft_consensus.cc:2229] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:20.179742 18207 raft_consensus.cc:2258] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.181612 18209 ts_tablet_manager.cc:1905] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:20.183314 18209 ts_tablet_manager.cc:1918] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:20.183393 18209 log.cc:1196] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/b301dbd97165489aae0804e1d349c6b3
I20260809 16:43:20.183730 18209 ts_tablet_manager.cc:1939] T b301dbd97165489aae0804e1d349c6b3 P 41cd206f2fa24f258261ccc887cbbda7: Deleting consensus metadata
I20260809 16:43:20.184525 17608 catalog_manager.cc:4841] TS 41cd206f2fa24f258261ccc887cbbda7 (127.28.62.131:37447): tablet b301dbd97165489aae0804e1d349c6b3 (table test-table [id=5605904091d347e3a2d28ff2db75ea1a]) successfully deleted
I20260809 16:43:20.185691 18207 ts_tablet_manager.cc:1905] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:20.187510 18207 ts_tablet_manager.cc:1918] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:20.187604 18207 log.cc:1196] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/b301dbd97165489aae0804e1d349c6b3
I20260809 16:43:20.187940 18207 ts_tablet_manager.cc:1939] T b301dbd97165489aae0804e1d349c6b3 P d49190ef2dc14476b3fc870c28cbb72d: Deleting consensus metadata
I20260809 16:43:20.188401 17610 catalog_manager.cc:4841] TS d49190ef2dc14476b3fc870c28cbb72d (127.28.62.130:37333): tablet b301dbd97165489aae0804e1d349c6b3 (table test-table [id=5605904091d347e3a2d28ff2db75ea1a]) successfully deleted
W20260809 16:43:20.190249 17610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:34273 (error 108)
I20260809 16:43:20.190601 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17861
I20260809 16:43:20.193240 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.190711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34273 (local address 127.0.0.1:54294)
0809 16:43:20.193139 (+ 2428us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.193195 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2401}
W20260809 16:43:20.193418 17610 catalog_manager.cc:4584] TS 1ac3e1cf62a04278a3ac740f3e7d3423 (127.28.62.129:34273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b301dbd97165489aae0804e1d349c6b3: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34273: connect: Connection refused (error 111)
I20260809 16:43:20.199949 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18015
I20260809 16:43:20.210000 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 17587
2026-08-09T16:43:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2100 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T16:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:20Z Disabled control of system clock
I20260809 16:43:20.241180 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46209
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:20.322592 18214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:20.322824 18214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:20.322846 18214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:20.324021 18214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:20.324066 18214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:20.324079 18214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:20.324092 18214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:20.324105 18214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:20.325608 18214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:46209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:20.325814 18214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:20.326054 18214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:20.330924 18219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.330919 18223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.330914 18220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.331336 18214 server_base.cc:1034] running on GCE node
I20260809 16:43:20.331604 18214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:20.331898 18214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:20.333099 18214 hybrid_clock.cc:648] HybridClock initialized: now 1786293800333038 us; error 49 us; skew 500 ppm
I20260809 16:43:20.334501 18214 webserver.cc:460] Webserver started at http://127.28.62.190:35635/ using document root <none> and password file <none>
I20260809 16:43:20.334779 18214 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:20.334833 18214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:20.334980 18214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:20.336167 18214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "9ec555c8b9174fd0adbd390195808b53"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "f4827cff72c653d7d515ac902a98c2ba"
server_key_iv: "cb0c40a6ad9f9bb7e6ab59ea4698a6ca"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.336545 18214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "9ec555c8b9174fd0adbd390195808b53"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "f4827cff72c653d7d515ac902a98c2ba"
server_key_iv: "cb0c40a6ad9f9bb7e6ab59ea4698a6ca"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.338006 18214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:20.338929 18229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.339165 18214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:20.339283 18214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "9ec555c8b9174fd0adbd390195808b53"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "f4827cff72c653d7d515ac902a98c2ba"
server_key_iv: "cb0c40a6ad9f9bb7e6ab59ea4698a6ca"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.339367 18214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:20.350005 18214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:20.350325 18214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:20.350492 18214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:20.354641 18214 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:46209
I20260809 16:43:20.354681 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:46209 every 8 connection(s)
I20260809 16:43:20.355124 18214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:20.355922 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18214
I20260809 16:43:20.356038 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:20.356254 18291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.356348 28922 external_mini_cluster.cc:1428] Setting key dea856d558ec79fdff3f86ba00b2e890
I20260809 16:43:20.359756 18291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: Bootstrap starting.
I20260809 16:43:20.360404 18291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.360741 18291 log.cc:824] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.361515 18291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: No bootstrap required, opened a new log
I20260809 16:43:20.362973 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.356944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.28.62.190:46209)
0809 16:43:20.357172 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.357180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.357225 (+ 45us) server_negotiation.cc:407] Connection header received
0809 16:43:20.357849 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.357858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.357938 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.358090 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.358334 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.359692 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.360760 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.361341 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.361464 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.362262 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.362267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.362267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:20.362473 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.362518 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:20.362531 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.362601 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.362679 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.362757 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260809 16:43:20.364111 18291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } }
I20260809 16:43:20.364243 18291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.364266 18291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec555c8b9174fd0adbd390195808b53, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.364385 18291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } }
I20260809 16:43:20.364462 18291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:20.364485 18291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:20.364513 18291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.365012 18291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } }
I20260809 16:43:20.365095 18291 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec555c8b9174fd0adbd390195808b53; no voters:
I20260809 16:43:20.365236 18291 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:20.365377 18296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:20.365497 18291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:20.365551 18296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec555c8b9174fd0adbd390195808b53, State: Running, Role: LEADER
I20260809 16:43:20.365762 18296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } }
I20260809 16:43:20.366220 18297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec555c8b9174fd0adbd390195808b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } } }
I20260809 16:43:20.366303 18297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:20.366532 18307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:20.366740 18298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec555c8b9174fd0adbd390195808b53. Latest consensus state: current_term: 1 leader_uuid: "9ec555c8b9174fd0adbd390195808b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec555c8b9174fd0adbd390195808b53" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 46209 } } }
I20260809 16:43:20.366858 18298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:20.367100 18307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:20.368664 18307 catalog_manager.cc:1347] Generated new cluster ID: c984a8fd2b4a46058db87171a9bccf8c
I20260809 16:43:20.368721 18307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:20.379691 18307 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:20.380426 18307 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:20.385372 18307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: Generated new TSK 0
I20260809 16:43:20.385552 18307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:20.391878 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--builtin_ntp_servers=127.28.62.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:20.474618 18319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:20.474884 18319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:20.474923 18319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:20.476189 18319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:20.476234 18319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:20.476260 18319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:20.477639 18319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:20.477921 18319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:20.478246 18319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:20.484207 18326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.484228 18319 server_base.cc:1034] running on GCE node
W20260809 16:43:20.484387 18325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.484197 18328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.484650 18319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:20.484911 18319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:20.486140 18319 hybrid_clock.cc:648] HybridClock initialized: now 1786293800486093 us; error 63 us; skew 500 ppm
I20260809 16:43:20.487723 18319 webserver.cc:460] Webserver started at http://127.28.62.129:43135/ using document root <none> and password file <none>
I20260809 16:43:20.488003 18319 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:20.488137 18319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:20.488322 18319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:20.489514 18319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "2e5614a779090fe02789f645e837e9c7"
server_key_iv: "d68ba8d3bbda2466e4f4930c2d807fe7"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.489951 18319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "2e5614a779090fe02789f645e837e9c7"
server_key_iv: "d68ba8d3bbda2466e4f4930c2d807fe7"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.491581 18319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260809 16:43:20.492503 18335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.492798 18319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:20.492901 18319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "2e5614a779090fe02789f645e837e9c7"
server_key_iv: "d68ba8d3bbda2466e4f4930c2d807fe7"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.492972 18319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:20.509433 18319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:20.509750 18319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:20.509865 18319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:20.510075 18319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:20.510459 18319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:20.510496 18319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.510521 18319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:20.510582 18319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.517373 18319 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:34167
I20260809 16:43:20.517454 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:34167 every 8 connection(s)
I20260809 16:43:20.517815 18319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:43:20.523466 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.518692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:37093 (local address 127.28.62.190:46209)
0809 16:43:20.518756 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.518759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.518989 (+ 230us) server_negotiation.cc:407] Connection header received
0809 16:43:20.519604 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.519605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.519622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.519691 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.520391 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.520905 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.521936 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.522146 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.522202 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.523162 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.523164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.523164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:20.523169 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.523182 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:20.523186 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.523242 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.523360 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.523389 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 16:43:20.523720 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.518636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46209 (local address 127.28.62.129:37093)
0809 16:43:20.518847 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.518890 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.519472 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.519785 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.519789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.519797 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:20.520305 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.520309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.521019 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.521020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.521843 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.521846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.522258 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.522259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.522370 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.522920 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:20.522921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:20.522949 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:20.522972 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:20.523274 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:20.523279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:20.523282 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.523341 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.523435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260809 16:43:20.524399 18476 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46209
I20260809 16:43:20.524498 18476 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:20.524717 18476 heartbeater.cc:511] Master 127.28.62.190:46209 requested a full tablet report, sending...
I20260809 16:43:20.525271 18249 ts_manager.cc:194] Registered new tserver with Master: e8a16fbf7c5a44408df4f0b1e4463e7e (127.28.62.129:34167)
I20260809 16:43:20.525882 18249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.129:37093
I20260809 16:43:20.528033 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18319
I20260809 16:43:20.528177 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:20.528340 28922 external_mini_cluster.cc:1428] Setting key 047c3e8d532325ca0da3dc6fc21dc3ed
I20260809 16:43:20.529291 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--builtin_ntp_servers=127.28.62.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:20.617846 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:20.618063 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:20.618086 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:20.619184 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:20.619218 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:20.619246 18479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:20.620517 18479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:20.620703 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:20.620961 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:20.626132 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.626132 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.626147 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.626180 18479 server_base.cc:1034] running on GCE node
I20260809 16:43:20.626543 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:20.626797 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:20.627964 18479 hybrid_clock.cc:648] HybridClock initialized: now 1786293800627909 us; error 72 us; skew 500 ppm
I20260809 16:43:20.629216 18479 webserver.cc:460] Webserver started at http://127.28.62.130:46127/ using document root <none> and password file <none>
I20260809 16:43:20.629503 18479 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:20.629577 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:20.629732 18479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:20.630812 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "f098dcf4e15a4761a94e394bc6702f3d"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "8b10b613b70158e71414a850e40576a6"
server_key_iv: "8573134ded1dec403e42b65232c3d35c"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.631248 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "f098dcf4e15a4761a94e394bc6702f3d"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "8b10b613b70158e71414a850e40576a6"
server_key_iv: "8573134ded1dec403e42b65232c3d35c"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.632741 18479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:20.633531 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.633728 18479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:20.633833 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "f098dcf4e15a4761a94e394bc6702f3d"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "8b10b613b70158e71414a850e40576a6"
server_key_iv: "8573134ded1dec403e42b65232c3d35c"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.633915 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:20.649643 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:20.650010 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:20.650175 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:20.650444 18479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:20.650792 18479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:20.650849 18479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.650887 18479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:20.650931 18479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.657061 18479 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:34601
I20260809 16:43:20.657114 18627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:34601 every 8 connection(s)
I20260809 16:43:20.657447 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:20.662803 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.658257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:48015 (local address 127.28.62.190:46209)
0809 16:43:20.658334 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.658348 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.658573 (+ 225us) server_negotiation.cc:407] Connection header received
0809 16:43:20.659105 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.659106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.659119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.659188 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.659873 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.660384 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.661349 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.661574 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.661646 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.662518 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.662519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.662519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:20.662524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.662538 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:20.662542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.662596 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.662711 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.662739 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:43:20.662904 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.658225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46209 (local address 127.28.62.130:48015)
0809 16:43:20.658447 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.658481 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.658960 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.659267 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.659270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.659279 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:20.659781 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.659785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.660469 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.660469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.661241 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.661243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.661666 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.661667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.661780 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.662370 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:20.662370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:20.662398 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:20.662420 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:20.662608 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:20.662612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:20.662614 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.662666 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.662752 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260809 16:43:20.663429 18628 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46209
I20260809 16:43:20.663508 18628 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:20.663710 18628 heartbeater.cc:511] Master 127.28.62.190:46209 requested a full tablet report, sending...
I20260809 16:43:20.664112 18249 ts_manager.cc:194] Registered new tserver with Master: f098dcf4e15a4761a94e394bc6702f3d (127.28.62.130:34601)
I20260809 16:43:20.664517 18249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.130:48015
I20260809 16:43:20.665261 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18479
I20260809 16:43:20.665391 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:20.665633 28922 external_mini_cluster.cc:1428] Setting key a13a9c399d2b72cd3e3e827ace2f5c8c
I20260809 16:43:20.666615 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--builtin_ntp_servers=127.28.62.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:20.749788 18631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:20.749998 18631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:20.750021 18631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:20.751211 18631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:20.751252 18631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:20.751277 18631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:20.752687 18631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:20.752943 18631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:20.753248 18631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:20.758356 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.758311 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:20.758307 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:20.758394 18631 server_base.cc:1034] running on GCE node
I20260809 16:43:20.758769 18631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:20.759083 18631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:20.760282 18631 hybrid_clock.cc:648] HybridClock initialized: now 1786293800760256 us; error 37 us; skew 500 ppm
I20260809 16:43:20.761622 18631 webserver.cc:460] Webserver started at http://127.28.62.131:42899/ using document root <none> and password file <none>
I20260809 16:43:20.761920 18631 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:20.762001 18631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:20.762220 18631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:20.763473 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "991b6e2b609743bab318e00bd344c8a5"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "44ef84d88722e13b31e63651fe62c582"
server_key_iv: "5d16066e270cfc29cabb69a7526fae84"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.763866 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "991b6e2b609743bab318e00bd344c8a5"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "44ef84d88722e13b31e63651fe62c582"
server_key_iv: "5d16066e270cfc29cabb69a7526fae84"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.765388 18631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:43:20.766299 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.766475 18631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:20.766582 18631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "991b6e2b609743bab318e00bd344c8a5"
format_stamp: "Formatted at 2026-08-09 16:43:20 on dist-test-slave-wl2h"
server_key: "44ef84d88722e13b31e63651fe62c582"
server_key_iv: "5d16066e270cfc29cabb69a7526fae84"
server_key_version: "encryptionkey@0"
I20260809 16:43:20.766672 18631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:20.790676 18631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:20.791105 18631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:20.791288 18631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:20.791554 18631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:20.791918 18631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:20.791977 18631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.792022 18631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:20.792088 18631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:20.798362 18631 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:36471
I20260809 16:43:20.798506 18782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:36471 every 8 connection(s)
I20260809 16:43:20.798810 18631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:20.802505 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18631
I20260809 16:43:20.802659 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:20.802865 28922 external_mini_cluster.cc:1428] Setting key 6ec5aef2ad08cb111bcc1c7bd448efa8
I20260809 16:43:20.804694 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.799726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:45109 (local address 127.28.62.190:46209)
0809 16:43:20.799804 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.799808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.800066 (+ 258us) server_negotiation.cc:407] Connection header received
0809 16:43:20.800571 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.800573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.800585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.800649 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.801373 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.801873 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.803080 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.803433 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.803510 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.804474 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.804475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.804476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:20.804480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.804494 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:20.804498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.804543 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.804612 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.804635 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 16:43:20.804966 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.799706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:46209 (local address 127.28.62.131:45109)
0809 16:43:20.799935 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.799971 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.800468 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.800712 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.800715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.800723 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:20.801239 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.801244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.802004 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.802005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.802929 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.802932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.803541 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.803542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.803701 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.804334 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:20.804335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:20.804363 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:20.804387 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:20.804556 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:20.804560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:20.804562 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.804613 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.804694 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":123,"threads_started":1}
I20260809 16:43:20.805621 18783 heartbeater.cc:347] Connected to a master server at 127.28.62.190:46209
I20260809 16:43:20.805751 18783 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:20.805945 18783 heartbeater.cc:511] Master 127.28.62.190:46209 requested a full tablet report, sending...
I20260809 16:43:20.806342 18249 ts_manager.cc:194] Registered new tserver with Master: 991b6e2b609743bab318e00bd344c8a5 (127.28.62.131:36471)
I20260809 16:43:20.806728 18249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.131:45109
I20260809 16:43:20.814610 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:20.819252 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.816202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.28.62.190:46209)
0809 16:43:20.816274 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.816276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.816321 (+ 45us) server_negotiation.cc:407] Connection header received
0809 16:43:20.816389 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.816389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.816398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.816449 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:20.816621 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.817104 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.817905 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.818101 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.818176 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.818911 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:20.818912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:20.818912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:20.818916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:20.818932 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:20.818935 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:20.819104 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.819180 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.819191 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 16:43:20.821107 18249 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:20.822676 18249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:20.829630 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.825159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.28.62.130:34601)
0809 16:43:20.825337 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.825343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.825353 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:20.825399 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.825401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.825423 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.825517 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.825702 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.827001 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.828330 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829106 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829189 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829289 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.829451 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.829534 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260809 16:43:20.829612 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.824945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:34601 (local address 127.0.0.1:33462)
0809 16:43:20.825158 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.825203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.825275 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.825523 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.825526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.825528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.825603 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.825615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.827132 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.827133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.828217 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.828220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829221 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.829320 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829337 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.829413 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.829475 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260809 16:43:20.829811 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.825215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.28.62.129:34167)
0809 16:43:20.825631 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.825636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.825646 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:20.825698 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.825700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.825714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.825806 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.825983 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.826953 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.828411 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829123 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829189 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829289 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.829653 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.829704 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":84,"threads_started":1}
I20260809 16:43:20.829823 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.825411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.28.62.131:36471)
0809 16:43:20.825937 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.825941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.825950 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:20.825999 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.826000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.826019 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.826097 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.826405 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.827507 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.828710 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829256 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829325 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829417 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.829664 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.829726 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":88,"threads_started":1}
I20260809 16:43:20.829877 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.824945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36471 (local address 127.0.0.1:39940)
0809 16:43:20.825364 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.825377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.825433 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.826243 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.826244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.826247 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.826306 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.826307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.827610 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.827611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.828603 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.828605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829454 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.829511 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829514 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.829565 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.829603 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":6,"spinlock_wait_cycles":768,"thread_start_us":96,"threads_started":1}
I20260809 16:43:20.829960 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.824955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34167 (local address 127.0.0.1:40706)
0809 16:43:20.825515 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.825525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.825563 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.825843 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.825844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.825846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.825887 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.825888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.827082 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.827083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.828326 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.828329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.829328 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.829329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.829427 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.829432 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.829488 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.829540 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260809 16:43:20.830690 18707 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 (DEFAULT_TABLE table=test-table [id=be970cce7605437a88c4d07c97016647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.830698 18398 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 (DEFAULT_TABLE table=test-table [id=be970cce7605437a88c4d07c97016647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.831084 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecce5532cbf4b7198b71e55a4eec973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.831110 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecce5532cbf4b7198b71e55a4eec973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.834101 18799 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Bootstrap starting.
I20260809 16:43:20.834255 18800 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: Bootstrap starting.
I20260809 16:43:20.834877 18800 tablet_bootstrap.cc:654] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.834915 18799 tablet_bootstrap.cc:654] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.834844 18549 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 (DEFAULT_TABLE table=test-table [id=be970cce7605437a88c4d07c97016647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:20.835268 18800 log.cc:824] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.835282 18549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecce5532cbf4b7198b71e55a4eec973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:20.835294 18799 log.cc:824] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.836045 18800 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: No bootstrap required, opened a new log
I20260809 16:43:20.836063 18799 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: No bootstrap required, opened a new log
I20260809 16:43:20.836138 18799 ts_tablet_manager.cc:1397] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 16:43:20.836135 18800 ts_tablet_manager.cc:1397] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:20.838680 18800 raft_consensus.cc:348] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.838855 18799 raft_consensus.cc:348] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.838965 18800 raft_consensus.cc:374] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.839061 18800 raft_consensus.cc:729] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a16fbf7c5a44408df4f0b1e4463e7e, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.839247 18799 raft_consensus.cc:374] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.839232 18800 consensus_queue.cc:260] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.839318 18799 raft_consensus.cc:729] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991b6e2b609743bab318e00bd344c8a5, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.839121 18803 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Bootstrap starting.
I20260809 16:43:20.839457 18799 consensus_queue.cc:260] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.839504 18800 ts_tablet_manager.cc:1428] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:20.839644 18476 heartbeater.cc:503] Master 127.28.62.190:46209 was elected leader, sending a full tablet report...
I20260809 16:43:20.840062 18799 ts_tablet_manager.cc:1428] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 16:43:20.840108 18783 heartbeater.cc:503] Master 127.28.62.190:46209 was elected leader, sending a full tablet report...
I20260809 16:43:20.840233 18803 tablet_bootstrap.cc:654] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:20.840588 18803 log.cc:824] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:20.841410 18803 tablet_bootstrap.cc:492] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: No bootstrap required, opened a new log
I20260809 16:43:20.841468 18803 ts_tablet_manager.cc:1397] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:20.843612 18803 raft_consensus.cc:348] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.843787 18803 raft_consensus.cc:374] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:20.843827 18803 raft_consensus.cc:729] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f098dcf4e15a4761a94e394bc6702f3d, State: Initialized, Role: FOLLOWER
I20260809 16:43:20.843945 18803 consensus_queue.cc:260] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.844223 18803 ts_tablet_manager.cc:1428] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:20.844343 18628 heartbeater.cc:503] Master 127.28.62.190:46209 was elected leader, sending a full tablet report...
I20260809 16:43:20.879617 18804 raft_consensus.cc:482] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:20.879765 18804 raft_consensus.cc:504] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.880192 18804 leader_election.cc:288] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f098dcf4e15a4761a94e394bc6702f3d (127.28.62.130:34601), 991b6e2b609743bab318e00bd344c8a5 (127.28.62.131:36471)
I20260809 16:43:20.883214 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.880555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:32949 (local address 127.28.62.130:34601)
0809 16:43:20.880601 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.880603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.880617 (+ 14us) server_negotiation.cc:407] Connection header received
0809 16:43:20.880649 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.880650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.880661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.880733 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.880935 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.881437 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882420 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.882810 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882866 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.882873 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.883081 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.883125 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:20.883200 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.880403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:34601 (local address 127.28.62.129:32949)
0809 16:43:20.880493 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.880508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.880573 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.880750 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.880751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.880753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.880835 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.880837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.881523 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.881524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.882322 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.882324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.882895 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.882976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.882980 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.883074 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.883120 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 16:43:20.883342 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.880564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:41749 (local address 127.28.62.131:36471)
0809 16:43:20.880627 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.880629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.880641 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:20.880755 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.880756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.880766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.880828 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:20.881041 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.881522 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882499 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.882903 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882975 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.882983 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.883220 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.883266 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:43:20.883406 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.880404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:36471 (local address 127.28.62.129:41749)
0809 16:43:20.880589 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.880599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:20.880655 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:20.880849 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:20.880850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:20.880852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:20.880950 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.880951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.881614 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.881615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.882391 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:20.882393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.882995 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.882996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:20.883128 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.883132 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 16:43:20.883202 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 16:43:20.883244 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":20,"thread_start_us":102,"threads_started":1}
I20260809 16:43:20.883632 18576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecce5532cbf4b7198b71e55a4eec973" candidate_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f098dcf4e15a4761a94e394bc6702f3d" is_pre_election: true
I20260809 16:43:20.883742 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecce5532cbf4b7198b71e55a4eec973" candidate_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991b6e2b609743bab318e00bd344c8a5" is_pre_election: true
I20260809 16:43:20.883822 18576 raft_consensus.cc:2454] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a16fbf7c5a44408df4f0b1e4463e7e in term 0.
I20260809 16:43:20.883939 18733 raft_consensus.cc:2454] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8a16fbf7c5a44408df4f0b1e4463e7e in term 0.
I20260809 16:43:20.884119 18337 leader_election.cc:302] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8a16fbf7c5a44408df4f0b1e4463e7e, f098dcf4e15a4761a94e394bc6702f3d; no voters:
I20260809 16:43:20.884303 18804 raft_consensus.cc:2781] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:20.884394 18804 raft_consensus.cc:482] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:20.884433 18804 raft_consensus.cc:3037] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.885167 18804 raft_consensus.cc:504] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.885331 18804 leader_election.cc:288] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [CANDIDATE]: Term 1 election: Requested vote from peers f098dcf4e15a4761a94e394bc6702f3d (127.28.62.130:34601), 991b6e2b609743bab318e00bd344c8a5 (127.28.62.131:36471)
I20260809 16:43:20.885483 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecce5532cbf4b7198b71e55a4eec973" candidate_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991b6e2b609743bab318e00bd344c8a5"
I20260809 16:43:20.885524 18576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecce5532cbf4b7198b71e55a4eec973" candidate_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f098dcf4e15a4761a94e394bc6702f3d"
I20260809 16:43:20.885564 18733 raft_consensus.cc:3037] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.885617 18576 raft_consensus.cc:3037] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:20.886436 18733 raft_consensus.cc:2454] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a16fbf7c5a44408df4f0b1e4463e7e in term 1.
I20260809 16:43:20.886600 18576 raft_consensus.cc:2454] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a16fbf7c5a44408df4f0b1e4463e7e in term 1.
I20260809 16:43:20.886654 18338 leader_election.cc:302] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 991b6e2b609743bab318e00bd344c8a5, e8a16fbf7c5a44408df4f0b1e4463e7e; no voters:
I20260809 16:43:20.886752 18804 raft_consensus.cc:2781] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:20.886919 18804 raft_consensus.cc:686] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 1 LEADER]: Becoming Leader. State: Replica: e8a16fbf7c5a44408df4f0b1e4463e7e, State: Running, Role: LEADER
I20260809 16:43:20.887060 18804 consensus_queue.cc:237] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } }
I20260809 16:43:20.887969 18248 catalog_manager.cc:5495] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e reported cstate change: term changed from 0 to 1, leader changed from <none> to e8a16fbf7c5a44408df4f0b1e4463e7e (127.28.62.129). New cstate: current_term: 1 leader_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8a16fbf7c5a44408df4f0b1e4463e7e" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 34167 } health_report { overall_health: HEALTHY } } }
I20260809 16:43:20.906243 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.903744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.28.62.129:34167)
0809 16:43:20.903823 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:20.903825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:20.903837 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:20.903896 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:20.903897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:20.903914 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:20.903996 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:20.904169 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.904638 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.905419 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:20.905615 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:20.905699 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:20.905868 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:20.905955 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:20.906074 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:20.906119 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 16:43:20.906128 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 16:43:20.908061 18629 tablet.cc:2369] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:20.908283 18576 raft_consensus.cc:1261] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 1 FOLLOWER]: Refusing update from remote peer e8a16fbf7c5a44408df4f0b1e4463e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:20.908311 18733 raft_consensus.cc:1261] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 1 FOLLOWER]: Refusing update from remote peer e8a16fbf7c5a44408df4f0b1e4463e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:20.908702 18804 consensus_queue.cc:1035] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f098dcf4e15a4761a94e394bc6702f3d" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:20.908910 18810 consensus_queue.cc:1035] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "991b6e2b609743bab318e00bd344c8a5" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 36471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:20.911608 18817 mvcc.cc:205] Tried to move back new op lower bound from 7316659408517259264 to 7316659408434106368. Current Snapshot: MvccSnapshot[applied={T|T < 7316659408517259264}]
I20260809 16:43:20.911831 18816 mvcc.cc:205] Tried to move back new op lower bound from 7316659408517259264 to 7316659408434106368. Current Snapshot: MvccSnapshot[applied={T|T < 7316659408517259264}]
I20260809 16:43:20.914273 18249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:20.914371 18249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:20.915284 18249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec555c8b9174fd0adbd390195808b53: Sending DeleteTablet for 3 replicas of tablet 7ecce5532cbf4b7198b71e55a4eec973
I20260809 16:43:20.915654 18549 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='slave'} at 127.0.0.1:33462
I20260809 16:43:20.915823 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='slave'} at 127.0.0.1:39940
I20260809 16:43:20.915910 18398 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecce5532cbf4b7198b71e55a4eec973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:20 UTC) from {username='slave'} at 127.0.0.1:40706
I20260809 16:43:20.915980 18826 tablet_replica.cc:330] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: stopping tablet replica
I20260809 16:43:20.916114 18829 tablet_replica.cc:330] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e: stopping tablet replica
I20260809 16:43:20.916090 18826 raft_consensus.cc:2229] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:20.916193 18829 raft_consensus.cc:2229] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:20.916235 18826 raft_consensus.cc:2258] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.916281 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18319
I20260809 16:43:20.916323 18829 raft_consensus.cc:2258] T 7ecce5532cbf4b7198b71e55a4eec973 P e8a16fbf7c5a44408df4f0b1e4463e7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.924516 18827 tablet_replica.cc:330] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: stopping tablet replica
I20260809 16:43:20.924875 18827 raft_consensus.cc:2229] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:20.925014 18827 raft_consensus.cc:2258] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:20.925336 18826 ts_tablet_manager.cc:1905] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:20.925565 18827 ts_tablet_manager.cc:1905] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:20.927006 18826 ts_tablet_manager.cc:1918] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:20.927127 18826 log.cc:1196] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/7ecce5532cbf4b7198b71e55a4eec973
I20260809 16:43:20.927394 18826 ts_tablet_manager.cc:1939] T 7ecce5532cbf4b7198b71e55a4eec973 P f098dcf4e15a4761a94e394bc6702f3d: Deleting consensus metadata
I20260809 16:43:20.927624 18827 ts_tablet_manager.cc:1918] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:20.927702 18827 log.cc:1196] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/7ecce5532cbf4b7198b71e55a4eec973
I20260809 16:43:20.927958 18827 ts_tablet_manager.cc:1939] T 7ecce5532cbf4b7198b71e55a4eec973 P 991b6e2b609743bab318e00bd344c8a5: Deleting consensus metadata
I20260809 16:43:20.927927 18230 catalog_manager.cc:4841] TS f098dcf4e15a4761a94e394bc6702f3d (127.28.62.130:34601): tablet 7ecce5532cbf4b7198b71e55a4eec973 (table test-table [id=be970cce7605437a88c4d07c97016647]) successfully deleted
I20260809 16:43:20.928387 18231 catalog_manager.cc:4841] TS 991b6e2b609743bab318e00bd344c8a5 (127.28.62.131:36471): tablet 7ecce5532cbf4b7198b71e55a4eec973 (table test-table [id=be970cce7605437a88c4d07c97016647]) successfully deleted
W20260809 16:43:20.929919 18232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:34167 (error 108)
I20260809 16:43:20.930488 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:20.930321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:34167 (local address 127.0.0.1:40710)
0809 16:43:20.930401 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:20.930451 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 16:43:20.930526 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18479
W20260809 16:43:20.940207 18232 catalog_manager.cc:4584] TS e8a16fbf7c5a44408df4f0b1e4463e7e (127.28.62.129:34167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ecce5532cbf4b7198b71e55a4eec973: Network error: Client connection negotiation failed: client connection to 127.28.62.129:34167: connect: Connection refused (error 111)
I20260809 16:43:20.946255 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18631
I20260809 16:43:20.955508 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18214
2026-08-09T16:43:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (746 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:21 dist-test-slave-wl2h krb5kdc[18833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:21 dist-test-slave-wl2h krb5kdc[18833](info): set up 2 sockets
Aug 09 16:43:21 dist-test-slave-wl2h krb5kdc[18833](info): commencing operation
krb5kdc: starting...
W20260809 16:43:22.105096 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:22.234589 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:39983
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:22.317050 18846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:22.317313 18846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:22.317350 18846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:22.318500 18846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:22.318575 18846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:22.318600 18846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:22.318622 18846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:22.318650 18846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:22.320182 18846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:22.320477 18846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:22.320729 18846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:22.325858 18860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.325834 18857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.325858 18858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:22.326119 18846 server_base.cc:1034] running on GCE node
I20260809 16:43:22.326411 18846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:22.326723 18846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:22.327914 18846 hybrid_clock.cc:648] HybridClock initialized: now 1786293802327901 us; error 56 us; skew 500 ppm
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:22.335768 18846 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:22.336355 18846 webserver.cc:460] Webserver started at http://127.28.62.190:40189/ using document root <none> and password file <none>
I20260809 16:43:22.336648 18846 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:22.336740 18846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:22.336920 18846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:22.338120 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "c82dadb1ec4449bcaee6413352fe3083"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "323ae4c41125ba9e14387f370fcd7f88"
server_key_iv: "8dd866ceb28278ba0f601c1194d9909a"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.338534 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "c82dadb1ec4449bcaee6413352fe3083"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "323ae4c41125ba9e14387f370fcd7f88"
server_key_iv: "8dd866ceb28278ba0f601c1194d9909a"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.340173 18846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260809 16:43:22.341050 18867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.341271 18846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:22.341398 18846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "c82dadb1ec4449bcaee6413352fe3083"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "323ae4c41125ba9e14387f370fcd7f88"
server_key_iv: "8dd866ceb28278ba0f601c1194d9909a"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.341481 18846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:22.361807 18846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:22.364497 18846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:22.364679 18846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:22.369349 18846 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:39983
I20260809 16:43:22.369421 18924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:39983 every 8 connection(s)
I20260809 16:43:22.369834 18846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:22.370682 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18846
I20260809 16:43:22.370834 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:22.371129 28922 external_mini_cluster.cc:1428] Setting key 1810ceee3b0f90b43e12551d25e755a2
I20260809 16:43:22.371292 18925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:22.374600 18925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: Bootstrap starting.
I20260809 16:43:22.375322 18925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:22.375685 18925 log.cc:824] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:22.376509 18925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: No bootstrap required, opened a new log
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:22.378435 18925 raft_consensus.cc:348] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } }
I20260809 16:43:22.378597 18925 raft_consensus.cc:374] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:22.378670 18925 raft_consensus.cc:729] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82dadb1ec4449bcaee6413352fe3083, State: Initialized, Role: FOLLOWER
I20260809 16:43:22.378842 18925 consensus_queue.cc:260] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } }
I20260809 16:43:22.378942 18925 raft_consensus.cc:388] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:22.378979 18925 raft_consensus.cc:482] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:22.379101 18925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:22.379662 18925 raft_consensus.cc:504] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } }
I20260809 16:43:22.379781 18925 leader_election.cc:302] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c82dadb1ec4449bcaee6413352fe3083; no voters:
I20260809 16:43:22.380002 18925 leader_election.cc:288] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:22.380030 18930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:22.380158 18930 raft_consensus.cc:686] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [term 1 LEADER]: Becoming Leader. State: Replica: c82dadb1ec4449bcaee6413352fe3083, State: Running, Role: LEADER
I20260809 16:43:22.380290 18925 sys_catalog.cc:564] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:22.380342 18930 consensus_queue.cc:237] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } }
I20260809 16:43:22.380771 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c82dadb1ec4449bcaee6413352fe3083. Latest consensus state: current_term: 1 leader_uuid: "c82dadb1ec4449bcaee6413352fe3083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } } }
I20260809 16:43:22.380784 18933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c82dadb1ec4449bcaee6413352fe3083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82dadb1ec4449bcaee6413352fe3083" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 39983 } } }
I20260809 16:43:22.380962 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:22.380975 18933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:22.381203 18940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:22.381806 18940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:22.382908 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.371743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.28.62.190:39983)
0809 16:43:22.371952 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.371961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.372015 (+ 54us) server_negotiation.cc:407] Connection header received
0809 16:43:22.372463 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.372469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.372516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.372633 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:22.373483 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.374399 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.375516 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.376149 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.376278 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.378752 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:22.378756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:22.378756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:22.378767 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:22.380931 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.381362 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.381365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.381365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.381466 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.381690 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.381691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.381691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.382180 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:22.382343 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.382441 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.382692 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":123,"threads_started":1}
I20260809 16:43:22.383378 18940 catalog_manager.cc:1347] Generated new cluster ID: ab6e31118ffd48a58d820123875d13c3
I20260809 16:43:22.383430 18940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:22.390431 18940 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:22.391171 18940 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:22.404770 18940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: Generated new TSK 0
I20260809 16:43:22.404990 18940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.62.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
Entry for principal kudu/127.28.62.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab.
I20260809 16:43:22.434602 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:22.517948 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:22.518148 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:22.518172 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:22.519263 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:22.519299 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:22.519325 18954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:22.520779 18954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.129.keytab
--principal=kudu/127.28.62.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:22.520965 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:22.521143 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:22.526106 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.526144 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.526325 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:22.526309 18954 server_base.cc:1034] running on GCE node
I20260809 16:43:22.526572 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:22.526767 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:22.527905 18954 hybrid_clock.cc:648] HybridClock initialized: now 1786293802527893 us; error 30 us; skew 500 ppm
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:22.530620 18954 init.cc:377] Logged in from keytab as kudu/127.28.62.129@KRBTEST.COM (short username kudu)
I20260809 16:43:22.531179 18954 webserver.cc:460] Webserver started at http://127.28.62.129:33083/ using document root <none> and password file <none>
I20260809 16:43:22.531427 18954 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:22.531497 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:22.531643 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:22.532699 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "29abbf73d7ba4ec2b813c5fbc79385b3"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "48ec7a2b2f22203b59f57cd0d2220ea3"
server_key_iv: "5553c463ee3bd7db80bbfebe5b802a0e"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.533066 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "29abbf73d7ba4ec2b813c5fbc79385b3"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "48ec7a2b2f22203b59f57cd0d2220ea3"
server_key_iv: "5553c463ee3bd7db80bbfebe5b802a0e"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.534462 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:22.535295 18970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.535480 18954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:22.535581 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "29abbf73d7ba4ec2b813c5fbc79385b3"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "48ec7a2b2f22203b59f57cd0d2220ea3"
server_key_iv: "5553c463ee3bd7db80bbfebe5b802a0e"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.535663 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:22.562844 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:22.565026 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:22.565214 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:22.565488 18954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:22.565824 18954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:22.565881 18954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.565917 18954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:22.565963 18954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.572418 18954 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:38227
I20260809 16:43:22.572474 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:38227 every 8 connection(s)
I20260809 16:43:22.572809 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.129@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:22.580451 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 18954
I20260809 16:43:22.580570 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:22.580783 28922 external_mini_cluster.cc:1428] Setting key 62c6500105080a1173df56faf8082489
I20260809 16:43:22.582620 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.573599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:37543 (local address 127.28.62.190:39983)
0809 16:43:22.573674 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.573675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.573867 (+ 192us) server_negotiation.cc:407] Connection header received
0809 16:43:22.574360 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.574361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.574372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.574433 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:22.575406 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.575946 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.576988 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.577202 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.577267 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.579449 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:22.579450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:22.579451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:22.579458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:22.581117 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.581650 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.581651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.581652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.581702 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.581986 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.581987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.581987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.582114 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:22.582178 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.582485 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.582555 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:43:22.582921 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.573573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:39983 (local address 127.28.62.129:37543)
0809 16:43:22.573726 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.573770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.574274 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.574483 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.574486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.575108 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:22.575305 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.575310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.576095 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.576096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.576899 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.576901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.577321 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.577322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.577436 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.577959 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:22.577967 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:22.579360 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:22.581211 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.581217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.581219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.581570 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.581775 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.581776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.581776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.581921 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.582179 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:22.582186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:22.582353 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.582466 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.582644 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260809 16:43:22.583506 19109 heartbeater.cc:347] Connected to a master server at 127.28.62.190:39983
I20260809 16:43:22.583629 19109 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:22.583830 19109 heartbeater.cc:511] Master 127.28.62.190:39983 requested a full tablet report, sending...
I20260809 16:43:22.584367 18886 ts_manager.cc:194] Registered new tserver with Master: 29abbf73d7ba4ec2b813c5fbc79385b3 (127.28.62.129:38227)
I20260809 16:43:22.584976 18886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.129@KRBTEST.COM'} at 127.28.62.129:37543
WARNING: no policy specified for kudu/127.28.62.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
Entry for principal kudu/127.28.62.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab.
I20260809 16:43:22.609676 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:22.691943 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:22.692231 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:22.692271 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:22.693398 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:22.693470 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:22.693513 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:22.694995 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.130.keytab
--principal=kudu/127.28.62.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:22.695297 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:22.695537 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:22.700680 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.700745 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.700672 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:22.700915 19114 server_base.cc:1034] running on GCE node
I20260809 16:43:22.701136 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:22.701434 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:22.702602 19114 hybrid_clock.cc:648] HybridClock initialized: now 1786293802702577 us; error 42 us; skew 500 ppm
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:22.705286 19114 init.cc:377] Logged in from keytab as kudu/127.28.62.130@KRBTEST.COM (short username kudu)
I20260809 16:43:22.705780 19114 webserver.cc:460] Webserver started at http://127.28.62.130:38065/ using document root <none> and password file <none>
I20260809 16:43:22.706012 19114 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:22.706084 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:22.706247 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:22.707384 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "9b2a60c6635744c2b38ac4922a0ea931"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "fab362229866395385ac58a3b3b1874b"
server_key_iv: "c0b99624b5d13a04445e9c94ac7233cc"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.707758 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "9b2a60c6635744c2b38ac4922a0ea931"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "fab362229866395385ac58a3b3b1874b"
server_key_iv: "c0b99624b5d13a04445e9c94ac7233cc"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.709105 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:22.709908 19132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.710088 19114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:43:22.710232 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "9b2a60c6635744c2b38ac4922a0ea931"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "fab362229866395385ac58a3b3b1874b"
server_key_iv: "c0b99624b5d13a04445e9c94ac7233cc"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.710315 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:22.723791 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:22.725725 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:22.725893 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:22.726140 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:22.726466 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:22.726522 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.726565 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:22.726591 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.732918 19114 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:43741
I20260809 16:43:22.732968 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:43741 every 8 connection(s)
I20260809 16:43:22.733289 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:22.735338 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19114
I20260809 16:43:22.735476 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:22.735677 28922 external_mini_cluster.cc:1428] Setting key d0994808b24c1379af867289999bad61
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.130@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:22.743784 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.734005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:52109 (local address 127.28.62.190:39983)
0809 16:43:22.734070 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.734072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.734282 (+ 210us) server_negotiation.cc:407] Connection header received
0809 16:43:22.734834 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.734835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.734850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.734917 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:22.736273 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.736783 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.737796 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.738006 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.738097 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.740717 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:22.740719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:22.740720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:22.740729 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:22.742260 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.742718 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.742719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.742720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.742780 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.743066 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.743067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.743067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.743226 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:22.743308 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.743591 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.743689 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:43:22.743981 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.733976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:39983 (local address 127.28.62.130:52109)
0809 16:43:22.734159 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.734196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.734715 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.734991 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.734994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.735909 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:22.736180 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.736185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.736903 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.736904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.737693 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.737695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.738110 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.738111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.738266 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.738854 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:22.738865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:22.740606 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:22.742378 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.742386 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.742389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.742623 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.742850 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.742851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.742851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.742966 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.743298 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:22.743306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:22.743450 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.743557 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.743738 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260809 16:43:22.744570 19266 heartbeater.cc:347] Connected to a master server at 127.28.62.190:39983
I20260809 16:43:22.744658 19266 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:22.744849 19266 heartbeater.cc:511] Master 127.28.62.190:39983 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.62.131@KRBTEST.COM; defaulting to no policy
I20260809 16:43:22.745250 18886 ts_manager.cc:194] Registered new tserver with Master: 9b2a60c6635744c2b38ac4922a0ea931 (127.28.62.130:43741)
I20260809 16:43:22.745630 18886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.130@KRBTEST.COM'} at 127.28.62.130:52109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
Entry for principal kudu/127.28.62.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab.
I20260809 16:43:22.765650 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:22.850404 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:22.850617 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:22.850641 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:22.851836 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:22.851886 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:22.851913 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:22.853434 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.131.keytab
--principal=kudu/127.28.62.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:22.853695 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:22.853971 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:22.859005 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.859006 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:22.859043 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:22.859445 19271 server_base.cc:1034] running on GCE node
I20260809 16:43:22.859687 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:22.859902 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:22.861054 19271 hybrid_clock.cc:648] HybridClock initialized: now 1786293802861027 us; error 41 us; skew 500 ppm
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:22.863788 19271 init.cc:377] Logged in from keytab as kudu/127.28.62.131@KRBTEST.COM (short username kudu)
I20260809 16:43:22.864267 19271 webserver.cc:460] Webserver started at http://127.28.62.131:36353/ using document root <none> and password file <none>
I20260809 16:43:22.864496 19271 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:22.864544 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:22.864643 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:22.865798 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "5ffd8912ac4c49868bd001914278a88b"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "30c84d2196fc758d2d5dc4b104cf79bf"
server_key_iv: "3ca9f1179afb94374f5b573ddb774209"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.866134 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "5ffd8912ac4c49868bd001914278a88b"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "30c84d2196fc758d2d5dc4b104cf79bf"
server_key_iv: "3ca9f1179afb94374f5b573ddb774209"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.867669 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:22.868448 19287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.868645 19271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:22.868743 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "5ffd8912ac4c49868bd001914278a88b"
format_stamp: "Formatted at 2026-08-09 16:43:22 on dist-test-slave-wl2h"
server_key: "30c84d2196fc758d2d5dc4b104cf79bf"
server_key_iv: "3ca9f1179afb94374f5b573ddb774209"
server_key_version: "encryptionkey@0"
I20260809 16:43:22.868803 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:22.884920 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:22.887400 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:22.887521 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:22.887712 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:22.888049 19271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:22.888084 19271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.888108 19271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:22.888154 19271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:22.894182 19271 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:37053
I20260809 16:43:22.894279 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:37053 every 8 connection(s)
I20260809 16:43:22.894615 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
Aug 09 16:43:22 dist-test-slave-wl2h krb5kdc[18833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.131@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:22.902169 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19271
I20260809 16:43:22.902299 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:22.902544 28922 external_mini_cluster.cc:1428] Setting key 1ae2670bbcd65fa70777ee9b2ee55395
I20260809 16:43:22.904654 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.895457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:48341 (local address 127.28.62.190:39983)
0809 16:43:22.895534 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.895536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.895767 (+ 231us) server_negotiation.cc:407] Connection header received
0809 16:43:22.896272 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.896273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.896284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.896345 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:22.897298 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.897823 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.898828 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.899097 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.899175 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.901490 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:22.901492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:22.901492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:22.901501 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:22.903233 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.903755 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.903756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.903756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.903808 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.904029 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.904030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.904030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.904135 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:22.904184 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.904452 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.904534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 16:43:22.904757 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.895445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:39983 (local address 127.28.62.131:48341)
0809 16:43:22.895631 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.895675 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.896159 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.896407 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.896411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.896979 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:22.897171 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.897176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.897934 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.897935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.898730 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.898732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.899195 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.899196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.899326 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.899835 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:22.899843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:22.901375 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:22.903322 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.903330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.903332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.903674 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.903870 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:22.903871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:22.903871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:22.903974 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:22.904189 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:22.904193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:22.904321 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.904404 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.904552 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260809 16:43:22.905159 19422 heartbeater.cc:347] Connected to a master server at 127.28.62.190:39983
I20260809 16:43:22.905254 19422 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:22.905447 19422 heartbeater.cc:511] Master 127.28.62.190:39983 requested a full tablet report, sending...
I20260809 16:43:22.905797 18886 ts_manager.cc:194] Registered new tserver with Master: 5ffd8912ac4c49868bd001914278a88b (127.28.62.131:37053)
I20260809 16:43:22.906198 18886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.62.131@KRBTEST.COM'} at 127.28.62.131:48341
I20260809 16:43:22.914565 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:22.922300 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.916436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.28.62.190:39983)
0809 16:43:22.916483 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.916484 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.916519 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:22.916575 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.916575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.916584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.916626 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:22.917180 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.917689 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.918421 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.918637 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.918702 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.920052 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:22.920053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:22.920054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:22.920061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:22.921278 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.921589 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.921590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.921590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.921642 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:22.921831 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:22.921832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:22.921832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:22.921980 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:22.922053 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.922154 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.922229 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:22.924000 18886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:22.925645 18886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:22.932598 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38227 (local address 127.0.0.1:45058)
0809 16:43:22.928764 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.928774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.928819 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.929340 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.929341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.929342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:22.929387 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.929388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930342 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.930343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.931368 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.931370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932282 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.932366 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932379 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.932453 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.932512 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260809 16:43:22.932624 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.28.62.129:38227)
0809 16:43:22.929183 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.929188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.929197 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:22.929238 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.929240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.929254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.929332 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:22.929471 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930210 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.931467 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932176 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932256 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932310 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.932483 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.932524 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":105,"threads_started":1}
I20260809 16:43:22.932812 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.28.62.130:43741)
0809 16:43:22.928549 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.928553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.928564 (+ 11us) server_negotiation.cc:407] Connection header received
0809 16:43:22.928610 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.928612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.928625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.928731 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:22.928912 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930095 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.931632 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932293 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932359 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932416 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.932645 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.932707 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260809 16:43:22.933056 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:37053 (local address 127.0.0.1:56604)
0809 16:43:22.928996 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.929007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.929051 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.929540 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.929541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.929543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:22.929572 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.929574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930760 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.930761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.931849 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.931851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932831 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.932904 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932909 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.932963 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.933003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":74,"threads_started":1}
I20260809 16:43:22.932935 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:43741 (local address 127.0.0.1:55244)
0809 16:43:22.928320 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:22.928362 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:22.928430 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:22.928734 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:22.928737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:22.928740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:22.928814 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.928816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930213 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.930215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.931537 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:22.931539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932450 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:22.932505 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932507 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:43:22.932553 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 16:43:22.932587 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":25,"thread_start_us":65,"threads_started":1}
I20260809 16:43:22.933313 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:22.928561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.28.62.131:37053)
0809 16:43:22.928742 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:22.928747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:22.929411 (+ 664us) server_negotiation.cc:407] Connection header received
0809 16:43:22.929454 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:22.929457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:22.929472 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:22.929543 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:22.929636 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.930643 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.931934 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:22.932651 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:22.932736 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:22.932790 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:22.933179 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 16:43:22.933228 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260809 16:43:22.933887 19030 tablet_service.cc:1467] Processing CreateTablet for tablet dce203db75bc4744b93e73a5ffd871fb (DEFAULT_TABLE table=test-table [id=2741de428f524bf98abbe2ecb4a9cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:22.933979 19342 tablet_service.cc:1467] Processing CreateTablet for tablet dce203db75bc4744b93e73a5ffd871fb (DEFAULT_TABLE table=test-table [id=2741de428f524bf98abbe2ecb4a9cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:22.934046 19187 tablet_service.cc:1467] Processing CreateTablet for tablet dce203db75bc4744b93e73a5ffd871fb (DEFAULT_TABLE table=test-table [id=2741de428f524bf98abbe2ecb4a9cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:22.934288 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce203db75bc4744b93e73a5ffd871fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:22.934346 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce203db75bc4744b93e73a5ffd871fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:22.934329 19342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce203db75bc4744b93e73a5ffd871fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:22.936945 19439 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Bootstrap starting.
I20260809 16:43:22.937642 19441 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Bootstrap starting.
I20260809 16:43:22.937963 19439 tablet_bootstrap.cc:654] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:22.938431 19441 tablet_bootstrap.cc:654] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:22.938627 19439 log.cc:824] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:22.939086 19441 log.cc:824] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:22.939209 19440 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Bootstrap starting.
I20260809 16:43:22.939538 19439 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: No bootstrap required, opened a new log
I20260809 16:43:22.939620 19439 ts_tablet_manager.cc:1397] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 16:43:22.939810 19441 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: No bootstrap required, opened a new log
I20260809 16:43:22.939836 19440 tablet_bootstrap.cc:654] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:22.939875 19441 ts_tablet_manager.cc:1397] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 16:43:22.940392 19440 log.cc:824] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:22.941094 19440 tablet_bootstrap.cc:492] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: No bootstrap required, opened a new log
I20260809 16:43:22.941179 19440 ts_tablet_manager.cc:1397] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 16:43:22.941615 19439 raft_consensus.cc:348] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.941824 19439 raft_consensus.cc:374] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:22.941869 19439 raft_consensus.cc:729] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffd8912ac4c49868bd001914278a88b, State: Initialized, Role: FOLLOWER
I20260809 16:43:22.941847 19441 raft_consensus.cc:348] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.942004 19441 raft_consensus.cc:374] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:22.941988 19439 consensus_queue.cc:260] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.942044 19441 raft_consensus.cc:729] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29abbf73d7ba4ec2b813c5fbc79385b3, State: Initialized, Role: FOLLOWER
I20260809 16:43:22.942148 19441 consensus_queue.cc:260] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.942294 19439 ts_tablet_manager.cc:1428] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:22.942432 19441 ts_tablet_manager.cc:1428] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 16:43:22.942589 19109 heartbeater.cc:503] Master 127.28.62.190:39983 was elected leader, sending a full tablet report...
I20260809 16:43:22.942692 19422 heartbeater.cc:503] Master 127.28.62.190:39983 was elected leader, sending a full tablet report...
I20260809 16:43:22.943085 19440 raft_consensus.cc:348] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.943267 19440 raft_consensus.cc:374] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:22.943311 19440 raft_consensus.cc:729] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b2a60c6635744c2b38ac4922a0ea931, State: Initialized, Role: FOLLOWER
I20260809 16:43:22.943427 19440 consensus_queue.cc:260] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:22.943691 19440 ts_tablet_manager.cc:1428] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 16:43:22.943773 19266 heartbeater.cc:503] Master 127.28.62.190:39983 was elected leader, sending a full tablet report...
W20260809 16:43:22.983873 19267 tablet.cc:2369] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 16:43:23.073648 19110 tablet.cc:2369] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:23.097149 19446 raft_consensus.cc:482] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:23.097327 19446 raft_consensus.cc:504] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:23.097800 19446 leader_election.cc:288] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29abbf73d7ba4ec2b813c5fbc79385b3 (127.28.62.129:38227), 9b2a60c6635744c2b38ac4922a0ea931 (127.28.62.130:43741)
I20260809 16:43:23.100996 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.098001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38227 (local address 127.28.62.131:41417)
0809 16:43:23.098065 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.098079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.098126 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.098264 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.098265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.098266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.098367 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.098370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.099243 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.099244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.100146 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.100149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.100727 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.100815 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.100820 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.100879 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.100924 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":25}
I20260809 16:43:23.101089 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.098039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41417 (local address 127.28.62.129:38227)
0809 16:43:23.098134 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.098137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.098170 (+ 33us) server_negotiation.cc:407] Connection header received
0809 16:43:23.098199 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.098200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.098210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.098260 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.098477 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.099152 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100251 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.100644 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100710 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.100716 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.100900 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.100944 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:23.101193 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.098125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:60193 (local address 127.28.62.130:43741)
0809 16:43:23.098180 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.098182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.098385 (+ 203us) server_negotiation.cc:407] Connection header received
0809 16:43:23.098420 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.098421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.098433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.098487 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.098698 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.099318 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100355 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.100784 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100864 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.100873 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.101061 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.101103 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260809 16:43:23.101296 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.098035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:43741 (local address 127.28.62.131:60193)
0809 16:43:23.098287 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.098297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.098374 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.098498 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.098499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.098500 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.098584 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.098586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.099421 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.099421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.100238 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.100242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.100894 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.100895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.100966 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.100971 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.101032 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.101068 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":129,"threads_started":1}
I20260809 16:43:23.101480 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce203db75bc4744b93e73a5ffd871fb" candidate_uuid: "5ffd8912ac4c49868bd001914278a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" is_pre_election: true
I20260809 16:43:23.101653 19054 raft_consensus.cc:2454] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ffd8912ac4c49868bd001914278a88b in term 0.
I20260809 16:43:23.101689 19210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce203db75bc4744b93e73a5ffd871fb" candidate_uuid: "5ffd8912ac4c49868bd001914278a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b2a60c6635744c2b38ac4922a0ea931" is_pre_election: true
I20260809 16:43:23.101874 19210 raft_consensus.cc:2454] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ffd8912ac4c49868bd001914278a88b in term 0.
I20260809 16:43:23.101930 19288 leader_election.cc:302] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29abbf73d7ba4ec2b813c5fbc79385b3, 5ffd8912ac4c49868bd001914278a88b; no voters:
I20260809 16:43:23.102147 19446 raft_consensus.cc:2781] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:23.102208 19446 raft_consensus.cc:482] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:23.102246 19446 raft_consensus.cc:3037] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.102969 19446 raft_consensus.cc:504] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:23.103173 19446 leader_election.cc:288] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [CANDIDATE]: Term 1 election: Requested vote from peers 29abbf73d7ba4ec2b813c5fbc79385b3 (127.28.62.129:38227), 9b2a60c6635744c2b38ac4922a0ea931 (127.28.62.130:43741)
I20260809 16:43:23.103354 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce203db75bc4744b93e73a5ffd871fb" candidate_uuid: "5ffd8912ac4c49868bd001914278a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3"
I20260809 16:43:23.103366 19210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce203db75bc4744b93e73a5ffd871fb" candidate_uuid: "5ffd8912ac4c49868bd001914278a88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b2a60c6635744c2b38ac4922a0ea931"
I20260809 16:43:23.103436 19054 raft_consensus.cc:3037] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.103479 19210 raft_consensus.cc:3037] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.104198 19054 raft_consensus.cc:2454] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffd8912ac4c49868bd001914278a88b in term 1.
I20260809 16:43:23.104291 19210 raft_consensus.cc:2454] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffd8912ac4c49868bd001914278a88b in term 1.
I20260809 16:43:23.104365 19288 leader_election.cc:302] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29abbf73d7ba4ec2b813c5fbc79385b3, 5ffd8912ac4c49868bd001914278a88b; no voters:
I20260809 16:43:23.104475 19446 raft_consensus.cc:2781] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:23.104689 19446 raft_consensus.cc:686] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 1 LEADER]: Becoming Leader. State: Replica: 5ffd8912ac4c49868bd001914278a88b, State: Running, Role: LEADER
I20260809 16:43:23.104802 19446 consensus_queue.cc:237] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } }
I20260809 16:43:23.105696 18885 catalog_manager.cc:5495] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ffd8912ac4c49868bd001914278a88b (127.28.62.131). New cstate: current_term: 1 leader_uuid: "5ffd8912ac4c49868bd001914278a88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ffd8912ac4c49868bd001914278a88b" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 37053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 } health_report { overall_health: UNKNOWN } } }
I20260809 16:43:23.121657 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.119163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.28.62.131:37053)
0809 16:43:23.119248 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.119250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.119259 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:23.119294 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.119295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.119312 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.119378 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:23.119503 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.119979 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.120750 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.120950 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.121015 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.121203 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:23.121312 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:23.121389 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.121478 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.121499 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 16:43:23.123713 19054 raft_consensus.cc:1261] T dce203db75bc4744b93e73a5ffd871fb P 29abbf73d7ba4ec2b813c5fbc79385b3 [term 1 FOLLOWER]: Refusing update from remote peer 5ffd8912ac4c49868bd001914278a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:23.123713 19210 raft_consensus.cc:1261] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 1 FOLLOWER]: Refusing update from remote peer 5ffd8912ac4c49868bd001914278a88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:23.124096 19450 consensus_queue.cc:1035] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29abbf73d7ba4ec2b813c5fbc79385b3" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:23.124243 19446 consensus_queue.cc:1035] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b2a60c6635744c2b38ac4922a0ea931" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:23.126556 19454 mvcc.cc:205] Tried to move back new op lower bound from 7316659417591603200 to 7316659417517850624. Current Snapshot: MvccSnapshot[applied={T|T < 7316659417591603200}]
I20260809 16:43:23.127162 19455 mvcc.cc:205] Tried to move back new op lower bound from 7316659417591603200 to 7316659417517850624. Current Snapshot: MvccSnapshot[applied={T|T < 7316659417591603200}]
I20260809 16:43:23.127177 19456 mvcc.cc:205] Tried to move back new op lower bound from 7316659417591603200 to 7316659417517850624. Current Snapshot: MvccSnapshot[applied={T|T < 7316659417591603200}]
I20260809 16:43:23.129112 18884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:23.129220 18884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:23.130131 18884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c82dadb1ec4449bcaee6413352fe3083: Sending DeleteTablet for 3 replicas of tablet dce203db75bc4744b93e73a5ffd871fb
I20260809 16:43:23.130494 19187 tablet_service.cc:1514] Processing DeleteTablet for tablet dce203db75bc4744b93e73a5ffd871fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:55244
I20260809 16:43:23.130501 19342 tablet_service.cc:1514] Processing DeleteTablet for tablet dce203db75bc4744b93e73a5ffd871fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:56604
I20260809 16:43:23.130787 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet dce203db75bc4744b93e73a5ffd871fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='kudu', principal='kudu/127.28.62.190@KRBTEST.COM'} at 127.0.0.1:45058
I20260809 16:43:23.130872 19464 tablet_replica.cc:330] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: stopping tablet replica
I20260809 16:43:23.130967 19464 raft_consensus.cc:2229] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:23.131146 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18954
I20260809 16:43:23.142241 19465 tablet_replica.cc:330] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: stopping tablet replica
I20260809 16:43:23.142416 19465 raft_consensus.cc:2229] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:23.142549 19465 raft_consensus.cc:2258] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:23.142728 19464 raft_consensus.cc:2258] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:23.143643 19464 ts_tablet_manager.cc:1905] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:23.145779 19464 ts_tablet_manager.cc:1918] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:23.145892 19464 log.cc:1196] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/dce203db75bc4744b93e73a5ffd871fb
I20260809 16:43:23.146173 19464 ts_tablet_manager.cc:1939] T dce203db75bc4744b93e73a5ffd871fb P 5ffd8912ac4c49868bd001914278a88b: Deleting consensus metadata
I20260809 16:43:23.146836 18868 catalog_manager.cc:4841] TS 5ffd8912ac4c49868bd001914278a88b (127.28.62.131:37053): tablet dce203db75bc4744b93e73a5ffd871fb (table test-table [id=2741de428f524bf98abbe2ecb4a9cbcf]) successfully deleted
I20260809 16:43:23.147519 19465 ts_tablet_manager.cc:1905] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:23.149168 19465 ts_tablet_manager.cc:1918] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:23.149281 19465 log.cc:1196] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/dce203db75bc4744b93e73a5ffd871fb
I20260809 16:43:23.149554 19465 ts_tablet_manager.cc:1939] T dce203db75bc4744b93e73a5ffd871fb P 9b2a60c6635744c2b38ac4922a0ea931: Deleting consensus metadata
I20260809 16:43:23.150031 18869 catalog_manager.cc:4841] TS 9b2a60c6635744c2b38ac4922a0ea931 (127.28.62.130:43741): tablet dce203db75bc4744b93e73a5ffd871fb (table test-table [id=2741de428f524bf98abbe2ecb4a9cbcf]) successfully deleted
W20260809 16:43:23.153275 18868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:38227 (error 108)
I20260809 16:43:23.153499 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19114
I20260809 16:43:23.154623 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.153811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:38227 (local address 127.0.0.1:45074)
0809 16:43:23.154493 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.154576 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652}
W20260809 16:43:23.159708 18868 catalog_manager.cc:4584] TS 29abbf73d7ba4ec2b813c5fbc79385b3 (127.28.62.129:38227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dce203db75bc4744b93e73a5ffd871fb: Network error: Client connection negotiation failed: client connection to 127.28.62.129:38227: connect: Connection refused (error 111)
I20260809 16:43:23.165366 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19271
I20260809 16:43:23.173480 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 18846
2026-08-09T16:43:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2216 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T16:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:23Z Disabled control of system clock
I20260809 16:43:23.204773 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45497
--webserver_interface=127.28.62.190
--webserver_port=0
--builtin_ntp_servers=127.28.62.148:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:23.287189 19472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:23.287412 19472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:23.287432 19472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:23.288489 19472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:23.288523 19472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:23.288537 19472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:23.288548 19472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:23.288560 19472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:23.289824 19472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:45497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:23.290021 19472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:23.290252 19472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:23.295061 19479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:23.295135 19472 server_base.cc:1034] running on GCE node
W20260809 16:43:23.295090 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.295076 19480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:23.295650 19472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:23.295940 19472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:23.297181 19472 hybrid_clock.cc:648] HybridClock initialized: now 1786293803297155 us; error 50 us; skew 500 ppm
I20260809 16:43:23.298620 19472 webserver.cc:460] Webserver started at http://127.28.62.190:43391/ using document root <none> and password file <none>
I20260809 16:43:23.298885 19472 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:23.298961 19472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:23.299157 19472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:23.300271 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "bc32fd9890af4fab94b318c28b0b0d4a"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "03bdcf77822c3922165d15beb3a3f70f"
server_key_iv: "49032b156a7a6c8ce4aeea4a7bbc971c"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.300642 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "bc32fd9890af4fab94b318c28b0b0d4a"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "03bdcf77822c3922165d15beb3a3f70f"
server_key_iv: "49032b156a7a6c8ce4aeea4a7bbc971c"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.301997 19472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:23.302853 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.303098 19472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:23.303200 19472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "bc32fd9890af4fab94b318c28b0b0d4a"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "03bdcf77822c3922165d15beb3a3f70f"
server_key_iv: "49032b156a7a6c8ce4aeea4a7bbc971c"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.303306 19472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:23.324548 19472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:23.324932 19472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:23.325103 19472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:23.329201 19472 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:45497
I20260809 16:43:23.329276 19551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:45497 every 8 connection(s)
I20260809 16:43:23.329636 19472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:23.330761 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:23.333161 19554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: Bootstrap starting.
I20260809 16:43:23.333775 19554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:23.334079 19554 log.cc:824] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:23.334824 19554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: No bootstrap required, opened a new log
I20260809 16:43:23.336467 19554 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } }
I20260809 16:43:23.336620 19554 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:23.336644 19554 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc32fd9890af4fab94b318c28b0b0d4a, State: Initialized, Role: FOLLOWER
I20260809 16:43:23.336794 19554 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } }
I20260809 16:43:23.336869 19554 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:23.336890 19554 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:23.336936 19554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.337471 19554 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } }
I20260809 16:43:23.337585 19554 leader_election.cc:302] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc32fd9890af4fab94b318c28b0b0d4a; no voters:
I20260809 16:43:23.337765 19554 leader_election.cc:288] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:23.337860 19557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:23.338033 19554 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:23.338070 19557 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [term 1 LEADER]: Becoming Leader. State: Replica: bc32fd9890af4fab94b318c28b0b0d4a, State: Running, Role: LEADER
I20260809 16:43:23.338266 19557 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } }
I20260809 16:43:23.338712 19559 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc32fd9890af4fab94b318c28b0b0d4a. Latest consensus state: current_term: 1 leader_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } } }
I20260809 16:43:23.338815 19559 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:23.338816 19558 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32fd9890af4fab94b318c28b0b0d4a" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 45497 } } }
I20260809 16:43:23.338965 19558 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:23.339205 19566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:23.339735 19566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:23.339886 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19472
I20260809 16:43:23.340010 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:23.340217 28922 external_mini_cluster.cc:1428] Setting key 2997e55da80613083c773f949989dd25
I20260809 16:43:23.341780 19566 catalog_manager.cc:1347] Generated new cluster ID: 8085528e5dbf44cf9e90e92043a507cc
I20260809 16:43:23.341840 19566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:23.347136 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.340939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.28.62.190:45497)
0809 16:43:23.341145 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.341153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.341186 (+ 33us) server_negotiation.cc:407] Connection header received
0809 16:43:23.341299 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.341305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.341374 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.341761 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:23.341852 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.343256 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.344443 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.345094 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.345704 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.346164 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:23.346169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:23.346169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:23.346388 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:23.346437 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:23.346452 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:23.346704 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.346747 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.346841 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260809 16:43:23.354012 19566 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:23.354561 19566 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:23.357025 19566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: Generated new TSK 0
I20260809 16:43:23.357292 19566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:23.363754 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.129:0
--local_ip_for_outbound_sockets=127.28.62.129
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--builtin_ntp_servers=127.28.62.148:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 16:43:23.456801 19578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:23.457037 19578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:23.457062 19578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:23.458271 19578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:23.458315 19578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:23.458343 19578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.129
I20260809 16:43:23.460199 19578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.62.129
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.129
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:23.460517 19578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:23.460875 19578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:23.467067 19583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.467109 19584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.467273 19587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:23.467247 19578 server_base.cc:1034] running on GCE node
I20260809 16:43:23.467473 19578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:23.467767 19578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:23.468976 19578 hybrid_clock.cc:648] HybridClock initialized: now 1786293803468948 us; error 45 us; skew 500 ppm
I20260809 16:43:23.470552 19578 webserver.cc:460] Webserver started at http://127.28.62.129:41439/ using document root <none> and password file <none>
I20260809 16:43:23.470815 19578 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:23.470887 19578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:23.471091 19578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:23.472245 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data/instance:
uuid: "ddb99954eb7a46a695c67a878358c2ab"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "2004b27face762797029b356721f6488"
server_key_iv: "fef5902aa96a6b7c71dc9d238263fc3b"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.472621 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance:
uuid: "ddb99954eb7a46a695c67a878358c2ab"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "2004b27face762797029b356721f6488"
server_key_iv: "fef5902aa96a6b7c71dc9d238263fc3b"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.474011 19578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:23.474815 19593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.475082 19578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:23.475184 19578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal
uuid: "ddb99954eb7a46a695c67a878358c2ab"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "2004b27face762797029b356721f6488"
server_key_iv: "fef5902aa96a6b7c71dc9d238263fc3b"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.475271 19578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:23.499709 19578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:23.500072 19578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:23.500245 19578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:23.500515 19578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:23.500887 19578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:23.500946 19578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.500989 19578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:23.501019 19578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.507314 19578 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.129:46729
I20260809 16:43:23.507373 19729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.129:46729 every 8 connection(s)
I20260809 16:43:23.507757 19578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/data/info.pb
I20260809 16:43:23.512593 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19578
I20260809 16:43:23.512703 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-0/wal/instance
I20260809 16:43:23.512943 28922 external_mini_cluster.cc:1428] Setting key 0a2e985586cd48535a03997c58354ea2
I20260809 16:43:23.513159 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.508570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:41547 (local address 127.28.62.190:45497)
0809 16:43:23.508640 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.508642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.508871 (+ 229us) server_negotiation.cc:407] Connection header received
0809 16:43:23.509355 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.509356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.509370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.509418 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:23.510067 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.510569 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.511667 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.511856 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.511912 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.512822 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:23.512823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:23.512826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:23.512837 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:23.512857 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:23.512860 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:23.512909 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.513042 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.513072 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:43:23.513406 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.508560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45497 (local address 127.28.62.129:41547)
0809 16:43:23.508743 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.508781 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.509269 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.509477 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.509480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.509488 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:23.509992 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.509996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.510688 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.510689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.511578 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.511581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.511933 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.511933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.512039 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.512648 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:23.512648 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:23.512694 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:23.512726 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:23.512932 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:23.512939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:23.512942 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.513023 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.513135 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260809 16:43:23.513911 19730 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45497
I20260809 16:43:23.514016 19730 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:23.514101 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.130:0
--local_ip_for_outbound_sockets=127.28.62.130
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--builtin_ntp_servers=127.28.62.148:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 16:43:23.514241 19730 heartbeater.cc:511] Master 127.28.62.190:45497 requested a full tablet report, sending...
I20260809 16:43:23.514807 19508 ts_manager.cc:194] Registered new tserver with Master: ddb99954eb7a46a695c67a878358c2ab (127.28.62.129:46729)
I20260809 16:43:23.515550 19508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.129:41547
W20260809 16:43:23.598132 19733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:23.598343 19733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:23.598366 19733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:23.599615 19733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:23.599658 19733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:23.599684 19733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.130
I20260809 16:43:23.601025 19733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.62.130
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.130
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:23.601281 19733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:23.601599 19733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:23.606555 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.606562 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.606582 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:23.606963 19733 server_base.cc:1034] running on GCE node
I20260809 16:43:23.607230 19733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:23.607496 19733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:23.608661 19733 hybrid_clock.cc:648] HybridClock initialized: now 1786293803608645 us; error 28 us; skew 500 ppm
I20260809 16:43:23.609972 19733 webserver.cc:460] Webserver started at http://127.28.62.130:38331/ using document root <none> and password file <none>
I20260809 16:43:23.610244 19733 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:23.610318 19733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:23.610478 19733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:23.611637 19733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data/instance:
uuid: "0466f81703e246188f05f4aeaf3d17f1"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "a672da5c7136fa436e1d306439203e5b"
server_key_iv: "9493688d0402b0ec7cff13959cbf5560"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.611999 19733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance:
uuid: "0466f81703e246188f05f4aeaf3d17f1"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "a672da5c7136fa436e1d306439203e5b"
server_key_iv: "9493688d0402b0ec7cff13959cbf5560"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.613330 19733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:23.614032 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.614214 19733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:43:23.614312 19733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal
uuid: "0466f81703e246188f05f4aeaf3d17f1"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "a672da5c7136fa436e1d306439203e5b"
server_key_iv: "9493688d0402b0ec7cff13959cbf5560"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.614394 19733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:23.629460 19733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:23.629832 19733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:23.630002 19733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:23.630239 19733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:23.630577 19733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:23.630640 19733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.630682 19733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:23.630710 19733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.636833 19733 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.130:36579
I20260809 16:43:23.636886 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.130:36579 every 8 connection(s)
I20260809 16:43:23.637254 19733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/data/info.pb
I20260809 16:43:23.640097 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19733
I20260809 16:43:23.640199 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal/instance
I20260809 16:43:23.640404 28922 external_mini_cluster.cc:1428] Setting key 8c58f0765b1cd06944371a4e130a1471
I20260809 16:43:23.641389 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.62.131:0
--local_ip_for_outbound_sockets=127.28.62.131
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--builtin_ntp_servers=127.28.62.148:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 16:43:23.643745 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.638050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.130:50435 (local address 127.28.62.190:45497)
0809 16:43:23.638113 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.638119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.638399 (+ 280us) server_negotiation.cc:407] Connection header received
0809 16:43:23.638947 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.638949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.638963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.639121 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:23.639867 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.640721 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.641932 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.642270 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.642350 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.643401 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:23.643403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:23.643403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:23.643408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:23.643428 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:23.643434 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:23.643501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.643607 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.643642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 16:43:23.643857 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.637988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45497 (local address 127.28.62.130:50435)
0809 16:43:23.638274 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.638313 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.638791 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.639196 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.639200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.639208 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:23.639779 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.639784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.640826 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.640827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.641848 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.641851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.642395 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.642396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.642518 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.643212 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:23.643213 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:23.643265 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:23.643299 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:23.643508 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:23.643514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:23.643517 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.643572 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.643658 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":143,"threads_started":1}
I20260809 16:43:23.644444 19889 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45497
I20260809 16:43:23.644554 19889 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:23.644744 19889 heartbeater.cc:511] Master 127.28.62.190:45497 requested a full tablet report, sending...
I20260809 16:43:23.645129 19508 ts_manager.cc:194] Registered new tserver with Master: 0466f81703e246188f05f4aeaf3d17f1 (127.28.62.130:36579)
I20260809 16:43:23.645651 19508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.130:50435
W20260809 16:43:23.725001 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:23.725216 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:23.725238 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:23.726286 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:23.726320 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:23.726343 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.62.131
I20260809 16:43:23.727626 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.62.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.62.131
--webserver_port=0
--tserver_master_addrs=127.28.62.190:45497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.62.131
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:23.727816 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:23.728067 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:23.733301 19904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.733301 19902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:23.733290 19901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:23.733665 19893 server_base.cc:1034] running on GCE node
I20260809 16:43:23.733896 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:23.734156 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:23.735347 19893 hybrid_clock.cc:648] HybridClock initialized: now 1786293803735330 us; error 33 us; skew 500 ppm
I20260809 16:43:23.736699 19893 webserver.cc:460] Webserver started at http://127.28.62.131:43411/ using document root <none> and password file <none>
I20260809 16:43:23.736984 19893 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:23.737061 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:23.737226 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:23.738757 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data/instance:
uuid: "74e9686083bf4df593427752379b17d6"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "8bdbfad033c631e2bc1086b9c0b21449"
server_key_iv: "cc1e03c44c6ff378cf5cf07c0abc33ab"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.739190 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance:
uuid: "74e9686083bf4df593427752379b17d6"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "8bdbfad033c631e2bc1086b9c0b21449"
server_key_iv: "cc1e03c44c6ff378cf5cf07c0abc33ab"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.740756 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:23.741523 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.741729 19893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:43:23.741842 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal
uuid: "74e9686083bf4df593427752379b17d6"
format_stamp: "Formatted at 2026-08-09 16:43:23 on dist-test-slave-wl2h"
server_key: "8bdbfad033c631e2bc1086b9c0b21449"
server_key_iv: "cc1e03c44c6ff378cf5cf07c0abc33ab"
server_key_version: "encryptionkey@0"
I20260809 16:43:23.741925 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:23.759882 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:23.760265 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:23.760442 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:23.760718 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 16:43:23.761087 19893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 16:43:23.761147 19893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.761189 19893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 16:43:23.761250 19893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:23.767683 19893 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.131:42479
I20260809 16:43:23.767767 20052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.131:42479 every 8 connection(s)
I20260809 16:43:23.768193 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/data/info.pb
I20260809 16:43:23.773715 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.769082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.131:41143 (local address 127.28.62.190:45497)
0809 16:43:23.769159 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.769162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.769406 (+ 244us) server_negotiation.cc:407] Connection header received
0809 16:43:23.769908 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.769909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.769919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.769970 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:23.770709 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.771281 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.772370 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.772559 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.772617 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.773427 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:23.773428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:23.773429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:23.773433 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:23.773447 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:23.773451 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:23.773496 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.773622 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.773648 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:43:23.773921 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.769055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:45497 (local address 127.28.62.131:41143)
0809 16:43:23.769275 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.769312 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.769819 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.770035 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.770038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.770046 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:23.770622 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.770626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.771370 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.771371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.772279 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.772282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.772644 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.772645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.772759 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.773283 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 16:43:23.773283 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:23.773312 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 16:43:23.773336 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:23.773532 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:23.773538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:23.773541 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.773617 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.773700 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260809 16:43:23.774438 20053 heartbeater.cc:347] Connected to a master server at 127.28.62.190:45497
I20260809 16:43:23.774531 20053 heartbeater.cc:464] Registering TS with master...
I20260809 16:43:23.774670 20053 heartbeater.cc:511] Master 127.28.62.190:45497 requested a full tablet report, sending...
I20260809 16:43:23.774994 19508 ts_manager.cc:194] Registered new tserver with Master: 74e9686083bf4df593427752379b17d6 (127.28.62.131:42479)
I20260809 16:43:23.775369 19508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.62.131:41143
I20260809 16:43:23.778306 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 19893
I20260809 16:43:23.778437 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal/instance
I20260809 16:43:23.778664 28922 external_mini_cluster.cc:1428] Setting key a1f1d0fa19ec1bc8963aac93ea983e63
I20260809 16:43:23.779778 28922 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 16:43:23.784475 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.781378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.28.62.190:45497)
0809 16:43:23.781469 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.781471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.781567 (+ 96us) server_negotiation.cc:407] Connection header received
0809 16:43:23.781657 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.781658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.781671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.781745 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:23.781924 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.782402 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.783267 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.783464 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.783553 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.784218 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:23.784219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:23.784219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:23.784225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:23.784241 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:23.784246 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:23.784301 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.784362 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.784387 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 16:43:23.786034 19508 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 16:43:23.787738 19508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 16:43:23.794848 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.28.62.130:36579)
0809 16:43:23.790975 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.790979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.790989 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:23.791118 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.791121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.791139 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.791215 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.791418 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.792125 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.793587 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.794455 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.794543 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.794644 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.794742 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.794776 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":175,"threads_started":1}
I20260809 16:43:23.794857 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36579 (local address 127.0.0.1:50158)
0809 16:43:23.790372 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.790419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.790497 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.791292 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.791293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.791294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.791331 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.791332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.792219 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.792220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.793494 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.793497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.794549 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.794550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.794634 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.794650 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.794726 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.794775 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":11,"thread_start_us":64,"threads_started":1}
I20260809 16:43:23.795257 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:46729 (local address 127.0.0.1:55634)
0809 16:43:23.790468 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.790479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.790529 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.790874 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.790877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.790880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.790958 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.790960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.792509 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.792510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.793565 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.793568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.794984 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.794985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.795079 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.795082 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.795138 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.795179 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":123,"threads_started":1}
I20260809 16:43:23.795506 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.28.62.129:46729)
0809 16:43:23.790645 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.790649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.790658 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:23.790705 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.790707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.790727 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.790828 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.791090 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.792415 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.793655 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.794455 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.794543 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.794645 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.795376 (+ 731us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.795423 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260809 16:43:23.795872 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:42479 (local address 127.0.0.1:41884)
0809 16:43:23.790620 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.790628 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.790673 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.790980 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.790981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.790983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.791057 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.791059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.793657 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.793657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.794304 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.794305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.795422 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.795423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.795486 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.795489 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.795531 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.795807 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":111,"threads_started":1}
I20260809 16:43:23.795883 19804 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ac13864a74b688b589153a27758ac (DEFAULT_TABLE table=test-table [id=685aba41eb724ff79dfc0414ed6c4952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:23.796180 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.790427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.28.62.131:42479)
0809 16:43:23.790638 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.790643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.790659 (+ 16us) server_negotiation.cc:407] Connection header received
0809 16:43:23.790759 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.790761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.790775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.790855 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.791153 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.792415 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.794393 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.795174 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.795235 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.796023 (+ 788us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.796056 (+ 33us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.796097 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
I20260809 16:43:23.796171 19649 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ac13864a74b688b589153a27758ac (DEFAULT_TABLE table=test-table [id=685aba41eb724ff79dfc0414ed6c4952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:23.796384 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ac13864a74b688b589153a27758ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:23.796533 19649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ac13864a74b688b589153a27758ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:23.796831 19972 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ac13864a74b688b589153a27758ac (DEFAULT_TABLE table=test-table [id=685aba41eb724ff79dfc0414ed6c4952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 16:43:23.797194 19972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ac13864a74b688b589153a27758ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:23.799103 20068 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: Bootstrap starting.
I20260809 16:43:23.799810 20068 tablet_bootstrap.cc:654] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:23.800086 20068 log.cc:824] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:23.800196 20069 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Bootstrap starting.
I20260809 16:43:23.800822 20069 tablet_bootstrap.cc:654] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:23.801070 20068 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: No bootstrap required, opened a new log
I20260809 16:43:23.801164 20068 ts_tablet_manager.cc:1397] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:23.801177 20069 log.cc:824] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:23.801513 20072 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Bootstrap starting.
I20260809 16:43:23.802330 20072 tablet_bootstrap.cc:654] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:23.802675 20072 log.cc:824] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:23.803472 20072 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: No bootstrap required, opened a new log
I20260809 16:43:23.803565 20072 ts_tablet_manager.cc:1397] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 16:43:23.803931 20069 tablet_bootstrap.cc:492] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: No bootstrap required, opened a new log
I20260809 16:43:23.804014 20069 ts_tablet_manager.cc:1397] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 16:43:23.804392 20068 raft_consensus.cc:348] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.804613 20068 raft_consensus.cc:374] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:23.804665 20068 raft_consensus.cc:729] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb99954eb7a46a695c67a878358c2ab, State: Initialized, Role: FOLLOWER
I20260809 16:43:23.804843 20068 consensus_queue.cc:260] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.805152 20068 ts_tablet_manager.cc:1428] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 16:43:23.805334 19730 heartbeater.cc:503] Master 127.28.62.190:45497 was elected leader, sending a full tablet report...
I20260809 16:43:23.806066 20072 raft_consensus.cc:348] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.806262 20072 raft_consensus.cc:374] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:23.806310 20072 raft_consensus.cc:729] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e9686083bf4df593427752379b17d6, State: Initialized, Role: FOLLOWER
I20260809 16:43:23.806457 20072 consensus_queue.cc:260] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.806767 20072 ts_tablet_manager.cc:1428] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 16:43:23.806937 20053 heartbeater.cc:503] Master 127.28.62.190:45497 was elected leader, sending a full tablet report...
I20260809 16:43:23.807228 20069 raft_consensus.cc:348] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.807411 20069 raft_consensus.cc:374] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:23.807462 20069 raft_consensus.cc:729] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0466f81703e246188f05f4aeaf3d17f1, State: Initialized, Role: FOLLOWER
I20260809 16:43:23.807595 20069 consensus_queue.cc:260] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.807912 20069 ts_tablet_manager.cc:1428] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 16:43:23.807973 19889 heartbeater.cc:503] Master 127.28.62.190:45497 was elected leader, sending a full tablet report...
W20260809 16:43:23.887851 19890 tablet.cc:2369] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 16:43:23.932420 20075 raft_consensus.cc:482] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 16:43:23.932559 20075 raft_consensus.cc:504] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.933001 20075 leader_election.cc:288] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0466f81703e246188f05f4aeaf3d17f1 (127.28.62.130:36579), 74e9686083bf4df593427752379b17d6 (127.28.62.131:42479)
I20260809 16:43:23.937175 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.933313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:59543 (local address 127.28.62.131:42479)
0809 16:43:23.933365 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.933367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.933529 (+ 162us) server_negotiation.cc:407] Connection header received
0809 16:43:23.933591 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.933592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.933602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.933659 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.933816 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.934548 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936214 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.936786 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936869 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.936877 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.937057 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.937103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 16:43:23.937215 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.933294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.62.129:39013 (local address 127.28.62.130:36579)
0809 16:43:23.933343 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.933345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.933357 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:23.933464 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.933465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.933473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.933550 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 16:43:23.933766 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.934548 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936206 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.936786 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936869 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.936877 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.937057 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.937144 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 16:43:23.937223 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.933209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.130:36579 (local address 127.28.62.129:39013)
0809 16:43:23.933287 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.933300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.933366 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.933569 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.933570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.933573 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.933683 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.933685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.934661 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.934663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.936080 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.936084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.936886 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.936983 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.936985 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.937042 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.937097 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":63}
I20260809 16:43:23.937237 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.933244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.131:42479 (local address 127.28.62.129:59543)
0809 16:43:23.933440 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:23.933454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:23.933507 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:23.933676 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:23.933676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:23.933678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 16:43:23.933745 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.933746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.934662 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.934663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.936083 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:23.936084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.936886 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.936888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:23.936973 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.936979 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 16:43:23.937042 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 16:43:23.937092 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":52,"thread_start_us":122,"threads_started":1}
I20260809 16:43:23.937664 19833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7ac13864a74b688b589153a27758ac" candidate_uuid: "ddb99954eb7a46a695c67a878358c2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0466f81703e246188f05f4aeaf3d17f1" is_pre_election: true
I20260809 16:43:23.937645 19997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7ac13864a74b688b589153a27758ac" candidate_uuid: "ddb99954eb7a46a695c67a878358c2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e9686083bf4df593427752379b17d6" is_pre_election: true
I20260809 16:43:23.937836 19833 raft_consensus.cc:2454] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb99954eb7a46a695c67a878358c2ab in term 0.
I20260809 16:43:23.937849 19997 raft_consensus.cc:2454] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb99954eb7a46a695c67a878358c2ab in term 0.
I20260809 16:43:23.938078 19596 leader_election.cc:302] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e9686083bf4df593427752379b17d6, ddb99954eb7a46a695c67a878358c2ab; no voters:
I20260809 16:43:23.938230 20075 raft_consensus.cc:2781] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 16:43:23.938300 20075 raft_consensus.cc:482] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 16:43:23.938337 20075 raft_consensus.cc:3037] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.939275 20075 raft_consensus.cc:504] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.939437 20075 leader_election.cc:288] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [CANDIDATE]: Term 1 election: Requested vote from peers 0466f81703e246188f05f4aeaf3d17f1 (127.28.62.130:36579), 74e9686083bf4df593427752379b17d6 (127.28.62.131:42479)
I20260809 16:43:23.939615 19997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7ac13864a74b688b589153a27758ac" candidate_uuid: "ddb99954eb7a46a695c67a878358c2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e9686083bf4df593427752379b17d6"
I20260809 16:43:23.939622 19833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7ac13864a74b688b589153a27758ac" candidate_uuid: "ddb99954eb7a46a695c67a878358c2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0466f81703e246188f05f4aeaf3d17f1"
I20260809 16:43:23.939721 19997 raft_consensus.cc:3037] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.939761 19833 raft_consensus.cc:3037] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:23.940595 19997 raft_consensus.cc:2454] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb99954eb7a46a695c67a878358c2ab in term 1.
I20260809 16:43:23.940759 19596 leader_election.cc:302] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e9686083bf4df593427752379b17d6, ddb99954eb7a46a695c67a878358c2ab; no voters:
I20260809 16:43:23.940761 19833 raft_consensus.cc:2454] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb99954eb7a46a695c67a878358c2ab in term 1.
I20260809 16:43:23.940866 20075 raft_consensus.cc:2781] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:23.941066 20075 raft_consensus.cc:686] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 1 LEADER]: Becoming Leader. State: Replica: ddb99954eb7a46a695c67a878358c2ab, State: Running, Role: LEADER
I20260809 16:43:23.941183 20075 consensus_queue.cc:237] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } }
I20260809 16:43:23.942021 19508 catalog_manager.cc:5495] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb99954eb7a46a695c67a878358c2ab (127.28.62.129). New cstate: current_term: 1 leader_uuid: "ddb99954eb7a46a695c67a878358c2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb99954eb7a46a695c67a878358c2ab" member_type: VOTER last_known_addr { host: "127.28.62.129" port: 46729 } health_report { overall_health: HEALTHY } } }
I20260809 16:43:23.981527 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:23.978912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.28.62.129:46729)
0809 16:43:23.978991 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:23.978992 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:23.978999 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:23.979085 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:23.979086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:23.979115 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:23.979214 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:23.979351 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.979838 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.980677 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:23.980877 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:23.980964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:23.981148 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:23.981258 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:23.981357 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:23.981402 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 16:43:23.981423 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 16:43:23.983690 19997 raft_consensus.cc:1261] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 1 FOLLOWER]: Refusing update from remote peer ddb99954eb7a46a695c67a878358c2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:23.983690 19833 raft_consensus.cc:1261] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 1 FOLLOWER]: Refusing update from remote peer ddb99954eb7a46a695c67a878358c2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:23.984054 20075 consensus_queue.cc:1035] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e9686083bf4df593427752379b17d6" member_type: VOTER last_known_addr { host: "127.28.62.131" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:23.984215 20079 consensus_queue.cc:1035] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0466f81703e246188f05f4aeaf3d17f1" member_type: VOTER last_known_addr { host: "127.28.62.130" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:23.986799 20082 mvcc.cc:205] Tried to move back new op lower bound from 7316659421113208832 to 7316659420943581184. Current Snapshot: MvccSnapshot[applied={T|T < 7316659421113208832}]
I20260809 16:43:23.987687 20084 mvcc.cc:205] Tried to move back new op lower bound from 7316659421113208832 to 7316659420943581184. Current Snapshot: MvccSnapshot[applied={T|T < 7316659421113208832 or (T in {7316659421113208832})}]
I20260809 16:43:23.988317 20083 mvcc.cc:205] Tried to move back new op lower bound from 7316659421113208832 to 7316659420943581184. Current Snapshot: MvccSnapshot[applied={T|T < 7316659421113208832}]
I20260809 16:43:23.989465 19507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:23.989566 19507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 16:43:23.990423 19507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc32fd9890af4fab94b318c28b0b0d4a: Sending DeleteTablet for 3 replicas of tablet 6a7ac13864a74b688b589153a27758ac
I20260809 16:43:23.990787 19649 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7ac13864a74b688b589153a27758ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='slave'} at 127.0.0.1:55634
I20260809 16:43:23.991082 20093 tablet_replica.cc:330] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab: stopping tablet replica
I20260809 16:43:23.991178 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19578
I20260809 16:43:23.991191 20093 raft_consensus.cc:2229] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 1 LEADER]: Raft consensus shutting down.
I20260809 16:43:23.991178 19972 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7ac13864a74b688b589153a27758ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='slave'} at 127.0.0.1:41884
I20260809 16:43:23.991310 20093 raft_consensus.cc:2258] T 6a7ac13864a74b688b589153a27758ac P ddb99954eb7a46a695c67a878358c2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:23.991530 19804 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7ac13864a74b688b589153a27758ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 16:43:23 UTC) from {username='slave'} at 127.0.0.1:50158
I20260809 16:43:23.991847 20096 tablet_replica.cc:330] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: stopping tablet replica
I20260809 16:43:23.991938 20096 raft_consensus.cc:2229] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:23.992027 20096 raft_consensus.cc:2258] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:23.995308 20095 tablet_replica.cc:330] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: stopping tablet replica
I20260809 16:43:23.995432 20095 raft_consensus.cc:2229] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 16:43:23.995498 20096 ts_tablet_manager.cc:1905] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:23.995544 20095 raft_consensus.cc:2258] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 16:43:23.996131 20095 ts_tablet_manager.cc:1905] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 16:43:23.997503 20096 ts_tablet_manager.cc:1918] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:23.997617 20096 log.cc:1196] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-1/wal/wals/6a7ac13864a74b688b589153a27758ac
I20260809 16:43:23.997862 20096 ts_tablet_manager.cc:1939] T 6a7ac13864a74b688b589153a27758ac P 0466f81703e246188f05f4aeaf3d17f1: Deleting consensus metadata
I20260809 16:43:23.998260 20095 ts_tablet_manager.cc:1918] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 16:43:23.998302 19492 catalog_manager.cc:4841] TS 0466f81703e246188f05f4aeaf3d17f1 (127.28.62.130:36579): tablet 6a7ac13864a74b688b589153a27758ac (table test-table [id=685aba41eb724ff79dfc0414ed6c4952]) successfully deleted
I20260809 16:43:23.998339 20095 log.cc:1196] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Deleting WAL directory at /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786293555794774-28922-0/minicluster-data/ts-2/wal/wals/6a7ac13864a74b688b589153a27758ac
I20260809 16:43:23.998579 20095 ts_tablet_manager.cc:1939] T 6a7ac13864a74b688b589153a27758ac P 74e9686083bf4df593427752379b17d6: Deleting consensus metadata
I20260809 16:43:23.999034 19493 catalog_manager.cc:4841] TS 74e9686083bf4df593427752379b17d6 (127.28.62.131:42479): tablet 6a7ac13864a74b688b589153a27758ac (table test-table [id=685aba41eb724ff79dfc0414ed6c4952]) successfully deleted
W20260809 16:43:24.000175 19491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.62.129:46729 (error 108)
I20260809 16:43:24.000396 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19733
I20260809 16:43:24.000763 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:24.000627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.129:46729 (local address 127.0.0.1:55650)
0809 16:43:24.000683 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:24.000725 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.129:46729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 16:43:24.000873 19491 catalog_manager.cc:4584] TS ddb99954eb7a46a695c67a878358c2ab (127.28.62.129:46729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a7ac13864a74b688b589153a27758ac: Network error: Client connection negotiation failed: client connection to 127.28.62.129:46729: connect: Connection refused (error 111)
I20260809 16:43:24.007125 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19893
I20260809 16:43:24.014739 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 19472
2026-08-09T16:43:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (842 ms)
[----------] 6 tests from EncryptionPolicyTest (8843 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:24 dist-test-slave-wl2h krb5kdc[20101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:24 dist-test-slave-wl2h krb5kdc[20101](info): set up 2 sockets
Aug 09 16:43:24 dist-test-slave-wl2h krb5kdc[20101](info): commencing operation
krb5kdc: starting...
W20260809 16:43:25.152690 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:25.282757 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:25.364697 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:25.364934 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:25.364972 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:25.366097 20119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:25.366166 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:25.366194 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:25.366223 20119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:25.366248 20119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:25.367784 20119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:25.368053 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:25.368281 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:25.373144 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:25.373157 20126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:25.373193 20129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:25.373257 20119 server_base.cc:1034] running on GCE node
I20260809 16:43:25.373651 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:25.373948 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:25.375072 20119 hybrid_clock.cc:648] HybridClock initialized: now 1786293805375066 us; error 39 us; skew 500 ppm
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:25.378527 20119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:25.379112 20119 webserver.cc:460] Webserver started at http://127.0.0.1:45633/ using document root <none> and password file <none>
I20260809 16:43:25.379382 20119 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:25.379452 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:25.379609 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:25.380949 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd"
format_stamp: "Formatted at 2026-08-09 16:43:25 on dist-test-slave-wl2h"
server_key: "83f7cced414a2d530e2563baa86ee2e7"
server_key_iv: "aa77a564ca86f3325feed6bcfd0d85fb"
server_key_version: "encryptionkey@0"
I20260809 16:43:25.381395 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd"
format_stamp: "Formatted at 2026-08-09 16:43:25 on dist-test-slave-wl2h"
server_key: "83f7cced414a2d530e2563baa86ee2e7"
server_key_iv: "aa77a564ca86f3325feed6bcfd0d85fb"
server_key_version: "encryptionkey@0"
I20260809 16:43:25.383240 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 16:43:25.384188 20137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:25.384375 20119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:25.384474 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd"
format_stamp: "Formatted at 2026-08-09 16:43:25 on dist-test-slave-wl2h"
server_key: "83f7cced414a2d530e2563baa86ee2e7"
server_key_iv: "aa77a564ca86f3325feed6bcfd0d85fb"
server_key_version: "encryptionkey@0"
I20260809 16:43:25.384543 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:25.396130 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:25.398116 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:25.398291 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:25.402280 20119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33867
I20260809 16:43:25.402331 20200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33867 every 8 connection(s)
I20260809 16:43:25.402700 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:25.403630 20201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:25.405869 20201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd: Bootstrap starting.
I20260809 16:43:25.406409 20201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:25.406658 20201 log.cc:824] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:25.407459 20201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd: No bootstrap required, opened a new log
I20260809 16:43:25.408129 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20119
I20260809 16:43:25.408258 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:25.408450 28922 external_mini_cluster.cc:1428] Setting key a9dde6c76b600779240f49908244c8cd
I20260809 16:43:25.409742 20201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260809 16:43:25.409876 20201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:25.409899 20201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4821cba12b4a0a82e02f7fc8dafdfd, State: Initialized, Role: FOLLOWER
I20260809 16:43:25.410027 20201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260809 16:43:25.410131 20201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:25.410168 20201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:25.410248 20201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:25.410998 20201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260809 16:43:25.411154 20201 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4821cba12b4a0a82e02f7fc8dafdfd; no voters:
I20260809 16:43:25.411361 20201 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:25.411394 20206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:25.411514 20206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [term 1 LEADER]: Becoming Leader. State: Replica: 4c4821cba12b4a0a82e02f7fc8dafdfd, State: Running, Role: LEADER
I20260809 16:43:25.411597 20206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260809 16:43:25.411625 20201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:25.412068 20208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4821cba12b4a0a82e02f7fc8dafdfd. Latest consensus state: current_term: 1 leader_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260809 16:43:25.412150 20208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:25.412679 20211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:25.413172 20211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:25.413538 20207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4821cba12b4a0a82e02f7fc8dafdfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260809 16:43:25.413702 20207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:25.414799 20211 catalog_manager.cc:1347] Generated new cluster ID: bb3fd05b92954b4fb01a23ea477ee837
I20260809 16:43:25.414849 20211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 16:43:25.420099 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:25.409001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38835 (local address 127.0.0.1:33867)
0809 16:43:25.409180 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:25.409187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:25.409213 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:43:25.409866 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:25.409896 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:25.409966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:25.410113 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:25.410932 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:25.412503 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:25.413679 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:25.414289 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:25.414410 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:25.416784 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:25.416787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:25.416788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:25.416798 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:25.418491 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:25.418892 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:25.418897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:25.418898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:25.418984 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:25.419276 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:25.419277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:25.419278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:25.419500 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:25.419709 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:25.419820 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:43:25.419928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260809 16:43:25.421248 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:25.428825 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:25.422710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37429 (local address 127.0.0.1:33867)
0809 16:43:25.422747 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:25.422749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:25.422865 (+ 116us) server_negotiation.cc:407] Connection header received
0809 16:43:25.422928 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:25.422928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:25.422938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:25.422983 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:25.423549 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:25.424024 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:25.424721 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:25.424950 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:25.425027 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:25.426201 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:25.426202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:25.426203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:25.426209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:25.427660 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:25.428026 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:25.428027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:25.428027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:25.428080 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:25.428314 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:25.428315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:25.428315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:25.428426 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:25.428535 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:25.428654 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 16:43:25.428713 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260809 16:43:25.440392 20211 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:25.440907 20211 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:25.443379 20211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4821cba12b4a0a82e02f7fc8dafdfd: Generated new TSK 0
I20260809 16:43:25.443568 20211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:25.448079 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20119
W20260809 16:43:25.458756 20098 connection.cc:535] client connection to 127.0.0.1:33867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33867): Connection reset by peer (error 104)
2026-08-09T16:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20238](info): set up 2 sockets
Aug 09 16:43:25 dist-test-slave-wl2h krb5kdc[20238](info): commencing operation
krb5kdc: starting...
W20260809 16:43:26.601135 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:26 dist-test-slave-wl2h krb5kdc[20238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:26.725354 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:26.808637 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:26.808851 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:26.808876 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:26.810036 20253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:26.810091 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:26.810104 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:26.810117 20253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:26.810164 20253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:26.811668 20253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:26.811920 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:26.812182 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:26.817096 20259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:26.817085 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:26.817085 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:26.817384 20253 server_base.cc:1034] running on GCE node
I20260809 16:43:26.817731 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:26.818027 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:26.819175 20253 hybrid_clock.cc:648] HybridClock initialized: now 1786293806819159 us; error 29 us; skew 500 ppm
Aug 09 16:43:26 dist-test-slave-wl2h krb5kdc[20238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:26.821710 20253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:26.822181 20253 webserver.cc:460] Webserver started at http://127.0.0.1:39705/ using document root <none> and password file <none>
I20260809 16:43:26.822443 20253 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:26.822511 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:26.822660 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:26.823807 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "3a6fdf41e93e42a388e66a35190e7ce9"
format_stamp: "Formatted at 2026-08-09 16:43:26 on dist-test-slave-wl2h"
server_key: "bb99ed40d4fd27667c9a177e1cd19449"
server_key_iv: "6cdf4ba634d537a7936920bc583cca94"
server_key_version: "encryptionkey@0"
I20260809 16:43:26.824177 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "3a6fdf41e93e42a388e66a35190e7ce9"
format_stamp: "Formatted at 2026-08-09 16:43:26 on dist-test-slave-wl2h"
server_key: "bb99ed40d4fd27667c9a177e1cd19449"
server_key_iv: "6cdf4ba634d537a7936920bc583cca94"
server_key_version: "encryptionkey@0"
I20260809 16:43:26.825594 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:26.826495 20273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:26.826699 20253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:26.826819 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "3a6fdf41e93e42a388e66a35190e7ce9"
format_stamp: "Formatted at 2026-08-09 16:43:26 on dist-test-slave-wl2h"
server_key: "bb99ed40d4fd27667c9a177e1cd19449"
server_key_iv: "6cdf4ba634d537a7936920bc583cca94"
server_key_version: "encryptionkey@0"
I20260809 16:43:26.826908 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:26.850170 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:26.852695 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:26.852977 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:26.857625 20253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34067
I20260809 16:43:26.857679 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34067 every 8 connection(s)
I20260809 16:43:26.858079 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:26.858953 20335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:26.860719 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20253
I20260809 16:43:26.860864 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:26.861079 28922 external_mini_cluster.cc:1428] Setting key 91b3c76afed70d4c56b03d5436fbbe63
I20260809 16:43:26.861693 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9: Bootstrap starting.
I20260809 16:43:26.862438 20335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:26.862792 20335 log.cc:824] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:26.863593 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9: No bootstrap required, opened a new log
I20260809 16:43:26.865908 20335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } }
I20260809 16:43:26.866111 20335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:26.866168 20335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6fdf41e93e42a388e66a35190e7ce9, State: Initialized, Role: FOLLOWER
I20260809 16:43:26.866338 20335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } }
I20260809 16:43:26.866463 20335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:26.866504 20335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:26.866578 20335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:26.867286 20335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } }
I20260809 16:43:26.867413 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:26.861686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57925 (local address 127.0.0.1:34067)
0809 16:43:26.861914 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:26.861922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:26.861957 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:26.862599 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:26.862609 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:26.862675 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:26.862856 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:26.863146 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:26.864157 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:26.865274 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:26.865663 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:26.865743 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:26.866638 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:26.866641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:26.866642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:26.866847 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:26.866877 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:26.866889 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:26.866956 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:26.867116 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 16:43:26.867157 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260809 16:43:26.867415 20335 leader_election.cc:302] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a6fdf41e93e42a388e66a35190e7ce9; no voters:
I20260809 16:43:26.867638 20335 leader_election.cc:288] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:26.867699 20342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:26.867841 20342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6fdf41e93e42a388e66a35190e7ce9, State: Running, Role: LEADER
I20260809 16:43:26.867882 20335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:26.867972 20342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } }
I20260809 16:43:26.868407 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:26.869871 20343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } } }
I20260809 16:43:26.870079 20343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:26.870338 20364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:26.870371 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a6fdf41e93e42a388e66a35190e7ce9. Latest consensus state: current_term: 1 leader_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6fdf41e93e42a388e66a35190e7ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34067 } } }
I20260809 16:43:26.870433 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:26.870874 20364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:26.872311 20364 catalog_manager.cc:1347] Generated new cluster ID: 612bd99e5f154c46af0d21974720cfc4
I20260809 16:43:26.872362 20364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:26.873641 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:26.869997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60261 (local address 127.0.0.1:34067)
0809 16:43:26.870058 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:26.870060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:26.870306 (+ 246us) server_negotiation.cc:407] Connection header received
0809 16:43:26.870376 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:26.870377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:26.870388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:26.870436 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:26.870628 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:26.871294 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:26.872302 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:26.872592 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:26.872635 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:26.873386 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:26.873387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:26.873387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:26.873391 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:26.873406 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:26.873410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:26.873463 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:26.873565 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:43:26.873574 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:43:26.879470 20364 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:26.880031 20364 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:26.899340 20364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a6fdf41e93e42a388e66a35190e7ce9: Generated new TSK 0
I20260809 16:43:26.899554 20364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:26.905745 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20253
W20260809 16:43:26.914572 20236 connection.cc:535] client connection to 127.0.0.1:34067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34067): Connection reset by peer (error 104)
2026-08-09T16:43:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:26 dist-test-slave-wl2h krb5kdc[20371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:26 dist-test-slave-wl2h krb5kdc[20371](info): set up 2 sockets
Aug 09 16:43:26 dist-test-slave-wl2h krb5kdc[20371](info): commencing operation
krb5kdc: starting...
W20260809 16:43:28.088397 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:28 dist-test-slave-wl2h krb5kdc[20371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:28.215790 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:28.297300 20387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:28.297505 20387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:28.297530 20387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:28.298667 20387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:28.298712 20387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:28.298725 20387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:28.298738 20387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:28.298779 20387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:28.300238 20387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:28.300493 20387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:28.300750 20387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:28.305775 20396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:28.305799 20394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:28.306010 20387 server_base.cc:1034] running on GCE node
W20260809 16:43:28.305786 20393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:28.306353 20387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:28.306666 20387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:28.307835 20387 hybrid_clock.cc:648] HybridClock initialized: now 1786293808307825 us; error 49 us; skew 500 ppm
Aug 09 16:43:28 dist-test-slave-wl2h krb5kdc[20371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:28.314365 20387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:28.314848 20387 webserver.cc:460] Webserver started at http://127.0.0.1:33069/ using document root <none> and password file <none>
I20260809 16:43:28.315105 20387 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:28.315156 20387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:28.315248 20387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:28.316288 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "9b43836d70aa41f384f3fbdaf786ba4f"
format_stamp: "Formatted at 2026-08-09 16:43:28 on dist-test-slave-wl2h"
server_key: "f932ae8e0b300d0e0c9121c94989e111"
server_key_iv: "35f3a647b95ccdf52808aa0b74225e56"
server_key_version: "encryptionkey@0"
I20260809 16:43:28.316601 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "9b43836d70aa41f384f3fbdaf786ba4f"
format_stamp: "Formatted at 2026-08-09 16:43:28 on dist-test-slave-wl2h"
server_key: "f932ae8e0b300d0e0c9121c94989e111"
server_key_iv: "35f3a647b95ccdf52808aa0b74225e56"
server_key_version: "encryptionkey@0"
I20260809 16:43:28.317914 20387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:28.318699 20406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:28.318928 20387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:28.319082 20387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "9b43836d70aa41f384f3fbdaf786ba4f"
format_stamp: "Formatted at 2026-08-09 16:43:28 on dist-test-slave-wl2h"
server_key: "f932ae8e0b300d0e0c9121c94989e111"
server_key_iv: "35f3a647b95ccdf52808aa0b74225e56"
server_key_version: "encryptionkey@0"
I20260809 16:43:28.319193 20387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:28.322471 20387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:28.324741 20387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:28.324854 20387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:28.328912 20387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42381
I20260809 16:43:28.328982 20469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42381 every 8 connection(s)
I20260809 16:43:28.329308 20387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:28.330323 20470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:28.331050 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20387
I20260809 16:43:28.331192 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:28.331382 28922 external_mini_cluster.cc:1428] Setting key d31884a4211a272426bb0be363a3cb3b
I20260809 16:43:28.333633 20470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: Bootstrap starting.
I20260809 16:43:28.334242 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:28.331930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40573 (local address 127.0.0.1:42381)
0809 16:43:28.332143 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:28.332153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:28.332188 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:28.332759 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:28.332768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:28.332803 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:28.332953 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:28.333637 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:28.333641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:28.333642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:28.333646 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:28.333692 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:28.333707 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:28.333781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:28.333998 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 16:43:28.334049 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260809 16:43:28.334375 20470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:28.334683 20470 log.cc:824] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:28.335338 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:28.335511 20470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: No bootstrap required, opened a new log
I20260809 16:43:28.337492 20470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260809 16:43:28.337661 20470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:28.337709 20470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b43836d70aa41f384f3fbdaf786ba4f, State: Initialized, Role: FOLLOWER
I20260809 16:43:28.337834 20470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260809 16:43:28.337932 20470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:28.337992 20470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:28.338059 20470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:28.338073 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:28.336876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33793 (local address 127.0.0.1:42381)
0809 16:43:28.336920 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:28.336922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:28.337048 (+ 126us) server_negotiation.cc:407] Connection header received
0809 16:43:28.337084 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:28.337086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:28.337092 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:28.337171 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:28.337780 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:28.337782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:28.337782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:28.337787 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:28.337802 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:28.337807 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:28.337837 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:28.337950 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 16:43:28.337964 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:28.338796 20470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260809 16:43:28.338918 20470 leader_election.cc:302] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b43836d70aa41f384f3fbdaf786ba4f; no voters:
I20260809 16:43:28.339103 20470 leader_election.cc:288] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:28.339205 20480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:28.339362 20480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [term 1 LEADER]: Becoming Leader. State: Replica: 9b43836d70aa41f384f3fbdaf786ba4f, State: Running, Role: LEADER
I20260809 16:43:28.339370 20470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:28.339493 20480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260809 16:43:28.339866 20481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } } }
I20260809 16:43:28.339876 20482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b43836d70aa41f384f3fbdaf786ba4f. Latest consensus state: current_term: 1 leader_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b43836d70aa41f384f3fbdaf786ba4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } } }
I20260809 16:43:28.340018 20481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:28.340024 20482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f [sys.catalog]: This master's current role is: LEADER
W20260809 16:43:28.341403 20497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:28.341446 20497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:28.341507 20489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:28.341663 20489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:28.343171 20489 catalog_manager.cc:1347] Generated new cluster ID: 75b929f8fb7b487085731e866b179a14
I20260809 16:43:28.343230 20489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:28.350764 20489 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:28.350927 20489 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:28.362795 20489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b43836d70aa41f384f3fbdaf786ba4f: Generated new TSK 0
I20260809 16:43:28.363054 20489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:28.372632 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20387
2026-08-09T16:43:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:28 dist-test-slave-wl2h krb5kdc[20502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:28 dist-test-slave-wl2h krb5kdc[20502](info): set up 2 sockets
Aug 09 16:43:28 dist-test-slave-wl2h krb5kdc[20502](info): commencing operation
krb5kdc: starting...
W20260809 16:43:29.521050 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:29.650202 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:29.734407 20516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:29.734674 20516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:29.734726 20516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:29.735918 20516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:29.735999 20516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:29.736025 20516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:29.736048 20516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:29.736076 20516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:29.737584 20516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:29.737869 20516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:29.738099 20516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:29.743013 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:29.743172 20516 server_base.cc:1034] running on GCE node
W20260809 16:43:29.743276 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:29.743059 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:29.743624 20516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:29.743916 20516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:29.745113 20516 hybrid_clock.cc:648] HybridClock initialized: now 1786293809745067 us; error 67 us; skew 500 ppm
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:29.747705 20516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:29.748209 20516 webserver.cc:460] Webserver started at http://127.0.0.1:39829/ using document root <none> and password file <none>
I20260809 16:43:29.748441 20516 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:29.748510 20516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:29.748667 20516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:29.749750 20516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "fca6a81b0db84f3fa99e21220fdfaa97"
format_stamp: "Formatted at 2026-08-09 16:43:29 on dist-test-slave-wl2h"
server_key: "85334d5645c79f2ecec2db2d3aca23d1"
server_key_iv: "4dce219b6841f24c98a8a9e01a75aa38"
server_key_version: "encryptionkey@0"
I20260809 16:43:29.750140 20516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "fca6a81b0db84f3fa99e21220fdfaa97"
format_stamp: "Formatted at 2026-08-09 16:43:29 on dist-test-slave-wl2h"
server_key: "85334d5645c79f2ecec2db2d3aca23d1"
server_key_iv: "4dce219b6841f24c98a8a9e01a75aa38"
server_key_version: "encryptionkey@0"
I20260809 16:43:29.751699 20516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:29.752571 20533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:29.752816 20516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:29.752946 20516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "fca6a81b0db84f3fa99e21220fdfaa97"
format_stamp: "Formatted at 2026-08-09 16:43:29 on dist-test-slave-wl2h"
server_key: "85334d5645c79f2ecec2db2d3aca23d1"
server_key_iv: "4dce219b6841f24c98a8a9e01a75aa38"
server_key_version: "encryptionkey@0"
I20260809 16:43:29.753036 20516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:29.767261 20516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:29.769917 20516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:29.770109 20516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:29.774729 20516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36611
I20260809 16:43:29.774806 20597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36611 every 8 connection(s)
I20260809 16:43:29.775249 20516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:29.776185 20598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:29.776481 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20516
I20260809 16:43:29.776618 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:29.776798 28922 external_mini_cluster.cc:1428] Setting key af19677c6fedb504e4e8f10710e009fb
I20260809 16:43:29.779750 20598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97: Bootstrap starting.
I20260809 16:43:29.780512 20598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:29.780833 20598 log.cc:824] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:29.781715 20598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97: No bootstrap required, opened a new log
I20260809 16:43:29.783779 20598 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260809 16:43:29.783964 20598 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:29.784001 20598 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca6a81b0db84f3fa99e21220fdfaa97, State: Initialized, Role: FOLLOWER
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 16:43:29.784124 20598 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260809 16:43:29.784196 20598 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:29.784232 20598 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:29.784289 20598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:29.784997 20598 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260809 16:43:29.785128 20598 leader_election.cc:302] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca6a81b0db84f3fa99e21220fdfaa97; no voters:
I20260809 16:43:29.785305 20598 leader_election.cc:288] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:29.785439 20603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:29.785593 20598 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:29.785665 20603 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [term 1 LEADER]: Becoming Leader. State: Replica: fca6a81b0db84f3fa99e21220fdfaa97, State: Running, Role: LEADER
I20260809 16:43:29.785830 20603 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260809 16:43:29.786288 20603 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca6a81b0db84f3fa99e21220fdfaa97. Latest consensus state: current_term: 1 leader_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260809 16:43:29.786341 20604 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca6a81b0db84f3fa99e21220fdfaa97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260809 16:43:29.786531 20604 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:29.786527 20603 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:29.786824 20616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:29.787416 20616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:29.789001 20616 catalog_manager.cc:1347] Generated new cluster ID: f7740eea60da4152acd719a3360a505a
I20260809 16:43:29.789074 20616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:29.789999 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:29.777338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41729 (local address 127.0.0.1:36611)
0809 16:43:29.777529 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:29.777537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:29.777579 (+ 42us) server_negotiation.cc:407] Connection header received
0809 16:43:29.778027 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:29.778033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:29.778081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:29.778197 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:29.779107 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:29.780074 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:29.781182 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:29.781755 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:29.781923 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:29.784756 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:29.784759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:29.784760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:29.784770 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:29.788115 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:29.788562 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:29.788565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:29.788566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:29.788767 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:29.788993 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:29.788994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:29.788994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:29.789334 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:29.789493 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:29.789616 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 16:43:29.789857 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260809 16:43:29.791893 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:29.801452 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:29.793424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36551 (local address 127.0.0.1:36611)
0809 16:43:29.793464 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:29.793466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:29.795238 (+ 1772us) server_negotiation.cc:407] Connection header received
0809 16:43:29.795317 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:29.795318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:29.795328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:29.795378 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:29.795933 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:29.796488 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:29.797257 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:29.797500 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:29.797595 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:29.798817 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:29.798818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:29.798819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:29.798825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:29.800291 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:29.800655 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:29.800656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:29.800657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:29.800712 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:29.800936 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:29.800936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:29.800937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:29.801051 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:29.801152 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:29.801261 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:43:29.801340 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:29.811924 20616 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:29.812708 20616 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:29.820817 20616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca6a81b0db84f3fa99e21220fdfaa97: Generated new TSK 0
I20260809 16:43:29.821014 20616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:29.827663 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20516
2026-08-09T16:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20634](info): set up 2 sockets
Aug 09 16:43:29 dist-test-slave-wl2h krb5kdc[20634](info): commencing operation
krb5kdc: starting...
W20260809 16:43:30.963347 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:31 dist-test-slave-wl2h krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:31.094009 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:31.177416 20649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:31.177644 20649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:31.177668 20649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:31.178957 20649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:31.179008 20649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:31.179088 20649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:31.179108 20649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:31.179139 20649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:31.180701 20649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:31.180968 20649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:31.181241 20649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:31.186470 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:31.186470 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:31.186499 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:31.186827 20649 server_base.cc:1034] running on GCE node
I20260809 16:43:31.187183 20649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:31.187501 20649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:31.188683 20649 hybrid_clock.cc:648] HybridClock initialized: now 1786293811188659 us; error 42 us; skew 500 ppm
Aug 09 16:43:31 dist-test-slave-wl2h krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:31.191604 20649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:31.192148 20649 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260809 16:43:31.192361 20649 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:31.192408 20649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:31.192507 20649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:31.193769 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "5e8e39eb2bc749d2a7ca78b8869e3636"
format_stamp: "Formatted at 2026-08-09 16:43:31 on dist-test-slave-wl2h"
server_key: "7153f4fc8401e8ccc93fe524715e5c2e"
server_key_iv: "adcb3510bb5c46132b2bdd872ef091ee"
server_key_version: "encryptionkey@0"
I20260809 16:43:31.194324 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "5e8e39eb2bc749d2a7ca78b8869e3636"
format_stamp: "Formatted at 2026-08-09 16:43:31 on dist-test-slave-wl2h"
server_key: "7153f4fc8401e8ccc93fe524715e5c2e"
server_key_iv: "adcb3510bb5c46132b2bdd872ef091ee"
server_key_version: "encryptionkey@0"
I20260809 16:43:31.196044 20649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260809 16:43:31.196972 20668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:31.197288 20649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:31.197373 20649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "5e8e39eb2bc749d2a7ca78b8869e3636"
format_stamp: "Formatted at 2026-08-09 16:43:31 on dist-test-slave-wl2h"
server_key: "7153f4fc8401e8ccc93fe524715e5c2e"
server_key_iv: "adcb3510bb5c46132b2bdd872ef091ee"
server_key_version: "encryptionkey@0"
I20260809 16:43:31.197432 20649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:31.207149 20649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:31.209374 20649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:31.209532 20649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:31.213896 20649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40445
I20260809 16:43:31.213932 20732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40445 every 8 connection(s)
I20260809 16:43:31.214295 20649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:31.215281 20734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:31.217532 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636: Bootstrap starting.
I20260809 16:43:31.218099 20734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:31.218335 20734 log.cc:824] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:31.219044 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636: No bootstrap required, opened a new log
I20260809 16:43:31.219638 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20649
I20260809 16:43:31.219774 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:31.219955 28922 external_mini_cluster.cc:1428] Setting key 5b79ded6ae2bc2e6e315cf0e5b747604
I20260809 16:43:31.221505 20734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260809 16:43:31.221714 20734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:31.221751 20734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e39eb2bc749d2a7ca78b8869e3636, State: Initialized, Role: FOLLOWER
I20260809 16:43:31.221886 20734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260809 16:43:31.221975 20734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:31.222009 20734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:31.222103 20734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:31.222860 20734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260809 16:43:31.222970 20734 leader_election.cc:302] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e8e39eb2bc749d2a7ca78b8869e3636; no voters:
I20260809 16:43:31.223155 20734 leader_election.cc:288] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:31.223233 20739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:31.223397 20734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:31.223443 20739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8e39eb2bc749d2a7ca78b8869e3636, State: Running, Role: LEADER
I20260809 16:43:31.223543 20739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260809 16:43:31.223893 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e8e39eb2bc749d2a7ca78b8869e3636. Latest consensus state: current_term: 1 leader_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } } }
I20260809 16:43:31.223979 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:31.223915 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8e39eb2bc749d2a7ca78b8869e3636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } } }
I20260809 16:43:31.224126 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:31.224946 20747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:31.225598 20747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:31.227203 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:31.220518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47293 (local address 127.0.0.1:40445)
0809 16:43:31.220735 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:31.220745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:31.220780 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:31.221728 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:31.221738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:31.221802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:31.221952 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:31.222277 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:31.223652 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:31.224883 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:31.225515 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:31.225635 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:31.226455 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:31.226459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:31.226460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:31.226690 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:31.226728 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:31.226743 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:31.226811 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:31.226891 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:43:31.226958 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260809 16:43:31.227566 20747 catalog_manager.cc:1347] Generated new cluster ID: 75124f0af45a43a4be9820f744324ca9
I20260809 16:43:31.227620 20747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:31.228188 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:31.233089 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:31.229632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53719 (local address 127.0.0.1:40445)
0809 16:43:31.229664 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:31.229666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:31.229786 (+ 120us) server_negotiation.cc:407] Connection header received
0809 16:43:31.229844 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:31.229845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:31.229854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:31.229912 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:31.230046 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:31.230731 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:31.231818 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:31.232101 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:31.232165 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:31.232856 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:31.232857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:31.232858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:31.232861 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:31.232875 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:31.232879 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:31.232923 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:31.232988 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 16:43:31.233009 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260809 16:43:31.234977 20747 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:31.235502 20747 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:31.242661 20747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e8e39eb2bc749d2a7ca78b8869e3636: Generated new TSK 0
I20260809 16:43:31.242861 20747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:31.247289 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20649
2026-08-09T16:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:31 dist-test-slave-wl2h krb5kdc[20766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:31 dist-test-slave-wl2h krb5kdc[20766](info): set up 2 sockets
Aug 09 16:43:31 dist-test-slave-wl2h krb5kdc[20766](info): commencing operation
krb5kdc: starting...
W20260809 16:43:32.411688 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:32 dist-test-slave-wl2h krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:32.538280 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:32.621652 20782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:32.621910 20782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:32.621953 20782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:32.623178 20782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:32.623240 20782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:32.623263 20782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:32.623283 20782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:32.623311 20782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:32.624805 20782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:32.625149 20782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:32.625404 20782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:32.630563 20787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:32.630604 20791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:32.630539 20789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:32.630752 20782 server_base.cc:1034] running on GCE node
I20260809 16:43:32.631068 20782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:32.631407 20782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:32.632560 20782 hybrid_clock.cc:648] HybridClock initialized: now 1786293812632539 us; error 40 us; skew 500 ppm
Aug 09 16:43:32 dist-test-slave-wl2h krb5kdc[20766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:32.635325 20782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:32.635790 20782 webserver.cc:460] Webserver started at http://127.0.0.1:42855/ using document root <none> and password file <none>
I20260809 16:43:32.636054 20782 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:32.636106 20782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:32.636255 20782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:32.637434 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "ebcf5004d8714606ab491a2f06417135"
format_stamp: "Formatted at 2026-08-09 16:43:32 on dist-test-slave-wl2h"
server_key: "c06616d8a433e62fb04f2ed62c3b7d87"
server_key_iv: "4a4952882ca98fa88f79c0ad24bf41bd"
server_key_version: "encryptionkey@0"
I20260809 16:43:32.637813 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "ebcf5004d8714606ab491a2f06417135"
format_stamp: "Formatted at 2026-08-09 16:43:32 on dist-test-slave-wl2h"
server_key: "c06616d8a433e62fb04f2ed62c3b7d87"
server_key_iv: "4a4952882ca98fa88f79c0ad24bf41bd"
server_key_version: "encryptionkey@0"
I20260809 16:43:32.639422 20782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:32.640228 20798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:32.640431 20782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:32.640524 20782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "ebcf5004d8714606ab491a2f06417135"
format_stamp: "Formatted at 2026-08-09 16:43:32 on dist-test-slave-wl2h"
server_key: "c06616d8a433e62fb04f2ed62c3b7d87"
server_key_iv: "4a4952882ca98fa88f79c0ad24bf41bd"
server_key_version: "encryptionkey@0"
I20260809 16:43:32.640579 20782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:32.643880 20782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:32.645999 20782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:32.646148 20782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:32.650105 20782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37353
I20260809 16:43:32.650183 20866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37353 every 8 connection(s)
I20260809 16:43:32.650493 20782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:32.651623 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:32.653429 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20782
I20260809 16:43:32.653555 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:32.653762 28922 external_mini_cluster.cc:1428] Setting key ea4c3cf28e19cc059a6504fc061157ad
I20260809 16:43:32.654229 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135: Bootstrap starting.
I20260809 16:43:32.654871 20867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:32.655210 20867 log.cc:824] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:32.656059 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135: No bootstrap required, opened a new log
I20260809 16:43:32.656383 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:32.654312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44405 (local address 127.0.0.1:37353)
0809 16:43:32.654516 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:32.654531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:32.654563 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:43:32.655094 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:32.655103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:32.655143 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:32.655288 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:32.655941 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:32.655944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:32.655944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:32.655948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:32.655984 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:32.655994 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:32.656069 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:32.656157 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 16:43:32.656235 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260809 16:43:32.657401 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:32.658789 20867 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260809 16:43:32.658958 20867 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:32.659005 20867 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcf5004d8714606ab491a2f06417135, State: Initialized, Role: FOLLOWER
I20260809 16:43:32.659170 20867 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260809 16:43:32.659286 20867 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:32.659324 20867 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:32.659374 20867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:32.659906 20867 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260809 16:43:32.660028 20867 leader_election.cc:302] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcf5004d8714606ab491a2f06417135; no voters:
I20260809 16:43:32.660208 20867 leader_election.cc:288] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:32.660255 20877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:32.660374 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:32.659315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40131 (local address 127.0.0.1:37353)
0809 16:43:32.659354 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:32.659356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:32.659362 (+ 6us) server_negotiation.cc:407] Connection header received
0809 16:43:32.659380 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:32.659381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:32.659385 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:32.659433 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:32.660049 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:32.660050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:32.660050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:32.660055 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:32.660067 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:32.660072 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:32.660122 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:32.660191 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 16:43:32.660203 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:32.660481 20877 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [term 1 LEADER]: Becoming Leader. State: Replica: ebcf5004d8714606ab491a2f06417135, State: Running, Role: LEADER
I20260809 16:43:32.660527 20867 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:32.660573 20877 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260809 16:43:32.660879 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcf5004d8714606ab491a2f06417135. Latest consensus state: current_term: 1 leader_uuid: "ebcf5004d8714606ab491a2f06417135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260809 16:43:32.661036 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:32.660893 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcf5004d8714606ab491a2f06417135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcf5004d8714606ab491a2f06417135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260809 16:43:32.661139 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:32.661299 20885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:32.661844 20885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:32.663357 20885 catalog_manager.cc:1347] Generated new cluster ID: a6089e3adac74e22a75d2516f06b41e7
I20260809 16:43:32.663398 20885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:32.675527 20885 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:32.675807 20885 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:32.687625 20885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcf5004d8714606ab491a2f06417135: Generated new TSK 0
I20260809 16:43:32.687872 20885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:32.690805 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20782
2026-08-09T16:43:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:32 dist-test-slave-wl2h krb5kdc[20902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:32 dist-test-slave-wl2h krb5kdc[20902](info): set up 2 sockets
Aug 09 16:43:32 dist-test-slave-wl2h krb5kdc[20902](info): commencing operation
krb5kdc: starting...
W20260809 16:43:33.813956 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:33 dist-test-slave-wl2h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:33.941902 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:34.025653 20917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:34.025867 20917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:34.025890 20917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:34.026949 20917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:34.026985 20917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:34.026999 20917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:34.027036 20917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:34.027050 20917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:34.028472 20917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:34.028664 20917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:34.028829 20917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:34.033815 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:34.033818 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:34.033835 20917 server_base.cc:1034] running on GCE node
W20260809 16:43:34.033854 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:34.034380 20917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:34.034673 20917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:34.035820 20917 hybrid_clock.cc:648] HybridClock initialized: now 1786293814035792 us; error 44 us; skew 500 ppm
Aug 09 16:43:34 dist-test-slave-wl2h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:34.043694 20917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:34.044211 20917 webserver.cc:460] Webserver started at http://127.0.0.1:41095/ using document root <none> and password file <none>
I20260809 16:43:34.044466 20917 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:34.044536 20917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:34.044696 20917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:34.045763 20917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "8c168824af124b2698fe9e4e91be72d9"
format_stamp: "Formatted at 2026-08-09 16:43:34 on dist-test-slave-wl2h"
server_key: "4c3ba93753c33c9a0399f4415bd0cfc8"
server_key_iv: "baa545840a9edcad9db0345ff0ccf507"
server_key_version: "encryptionkey@0"
I20260809 16:43:34.046133 20917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "8c168824af124b2698fe9e4e91be72d9"
format_stamp: "Formatted at 2026-08-09 16:43:34 on dist-test-slave-wl2h"
server_key: "4c3ba93753c33c9a0399f4415bd0cfc8"
server_key_iv: "baa545840a9edcad9db0345ff0ccf507"
server_key_version: "encryptionkey@0"
I20260809 16:43:34.047680 20917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:34.048530 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:34.048728 20917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:34.048836 20917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "8c168824af124b2698fe9e4e91be72d9"
format_stamp: "Formatted at 2026-08-09 16:43:34 on dist-test-slave-wl2h"
server_key: "4c3ba93753c33c9a0399f4415bd0cfc8"
server_key_iv: "baa545840a9edcad9db0345ff0ccf507"
server_key_version: "encryptionkey@0"
I20260809 16:43:34.048913 20917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:34.063877 20917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:34.065915 20917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:34.066079 20917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:34.070387 20917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33617
I20260809 16:43:34.070420 21001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33617 every 8 connection(s)
I20260809 16:43:34.070809 20917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:34.071806 21003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:34.074043 21003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9: Bootstrap starting.
I20260809 16:43:34.074570 21003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:34.074826 21003 log.cc:824] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:34.075544 21003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9: No bootstrap required, opened a new log
I20260809 16:43:34.077123 21003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260809 16:43:34.077270 21003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:34.077292 21003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c168824af124b2698fe9e4e91be72d9, State: Initialized, Role: FOLLOWER
I20260809 16:43:34.077517 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 20917
I20260809 16:43:34.077513 21003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260809 16:43:34.077600 21003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:34.077647 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:34.077657 21003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:34.077708 21003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:34.077863 28922 external_mini_cluster.cc:1428] Setting key 6611831d79e916b029b3de6b71fae5e2
I20260809 16:43:34.078365 21003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260809 16:43:34.078478 21003 leader_election.cc:302] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c168824af124b2698fe9e4e91be72d9; no voters:
I20260809 16:43:34.078645 21003 leader_election.cc:288] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:34.078678 21008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:34.078863 21008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8c168824af124b2698fe9e4e91be72d9, State: Running, Role: LEADER
I20260809 16:43:34.078907 21003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:34.078986 21008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } }
I20260809 16:43:34.079473 21010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c168824af124b2698fe9e4e91be72d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } } }
I20260809 16:43:34.079561 21010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:34.079721 21011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c168824af124b2698fe9e4e91be72d9. Latest consensus state: current_term: 1 leader_uuid: "8c168824af124b2698fe9e4e91be72d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168824af124b2698fe9e4e91be72d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33617 } } }
I20260809 16:43:34.079790 21011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:34.080158 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:34.080677 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:34.082432 21016 catalog_manager.cc:1347] Generated new cluster ID: c0d071e829934440b438c36eb8d77565
I20260809 16:43:34.082489 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:43:34 dist-test-slave-wl2h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 16:43:34.089381 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:34.078362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:50453 (local address 127.0.0.1:33617)
0809 16:43:34.078578 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:34.078588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:34.078622 (+ 34us) server_negotiation.cc:407] Connection header received
0809 16:43:34.078761 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:34.078784 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:34.078854 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:34.079928 (+ 1074us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:34.080040 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:34.081499 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:34.082526 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:34.082970 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:34.083140 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:34.085530 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:34.085534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:34.085534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:34.085546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:34.087815 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:34.088217 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:34.088222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:34.088223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:34.088321 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:34.088562 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:34.088563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:34.088564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:34.088809 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:34.088993 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:34.089105 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 16:43:34.089229 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260809 16:43:34.090291 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:34.101403 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:34.095333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:39159 (local address 127.0.0.1:33617)
0809 16:43:34.095375 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:34.095377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:34.095511 (+ 134us) server_negotiation.cc:407] Connection header received
0809 16:43:34.095581 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:34.095582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:34.095592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:34.095636 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:34.096144 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:34.096623 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:34.097322 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:34.097560 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:34.097633 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:34.098790 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:34.098791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:34.098791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:34.098799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:34.100217 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:34.100593 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:34.100595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:34.100595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:34.100676 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:34.100900 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:34.100901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:34.100902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:34.101009 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:34.101117 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:34.101239 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 16:43:34.101300 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:34.102912 21016 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:34.103614 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:34.110230 21016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c168824af124b2698fe9e4e91be72d9: Generated new TSK 0
I20260809 16:43:34.110411 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:34.116465 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 20917
W20260809 16:43:34.126924 20900 connection.cc:535] client connection to 127.0.0.1:33617 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33617): Connection reset by peer (error 104)
2026-08-09T16:43:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:34 dist-test-slave-wl2h krb5kdc[21042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:34 dist-test-slave-wl2h krb5kdc[21042](info): set up 2 sockets
Aug 09 16:43:34 dist-test-slave-wl2h krb5kdc[21042](info): commencing operation
krb5kdc: starting...
W20260809 16:43:35.254165 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:35 dist-test-slave-wl2h krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:35.382423 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:35.465972 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:35.466180 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:35.466214 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:35.467440 21056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:35.467483 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:35.467497 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:35.467510 21056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:35.467522 21056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:35.468993 21056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:35.469202 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:35.469383 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:35.474372 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:35.474509 21056 server_base.cc:1034] running on GCE node
W20260809 16:43:35.474382 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:35.474421 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:35.474951 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:35.475287 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:35.476501 21056 hybrid_clock.cc:648] HybridClock initialized: now 1786293815476496 us; error 46 us; skew 500 ppm
Aug 09 16:43:35 dist-test-slave-wl2h krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:35.479097 21056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:35.479588 21056 webserver.cc:460] Webserver started at http://127.0.0.1:37263/ using document root <none> and password file <none>
I20260809 16:43:35.479840 21056 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:35.479910 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:35.480054 21056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:35.481133 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "c3d77fb5835e45dfb197f3e6de2fa54d"
format_stamp: "Formatted at 2026-08-09 16:43:35 on dist-test-slave-wl2h"
server_key: "01e023cd69cada9d07b494dc28171303"
server_key_iv: "d5c7abcfe2fff23a5bc4b40b74b2a35a"
server_key_version: "encryptionkey@0"
I20260809 16:43:35.481566 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "c3d77fb5835e45dfb197f3e6de2fa54d"
format_stamp: "Formatted at 2026-08-09 16:43:35 on dist-test-slave-wl2h"
server_key: "01e023cd69cada9d07b494dc28171303"
server_key_iv: "d5c7abcfe2fff23a5bc4b40b74b2a35a"
server_key_version: "encryptionkey@0"
I20260809 16:43:35.482955 21056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:35.483758 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:35.483968 21056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:35.484079 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "c3d77fb5835e45dfb197f3e6de2fa54d"
format_stamp: "Formatted at 2026-08-09 16:43:35 on dist-test-slave-wl2h"
server_key: "01e023cd69cada9d07b494dc28171303"
server_key_iv: "d5c7abcfe2fff23a5bc4b40b74b2a35a"
server_key_version: "encryptionkey@0"
I20260809 16:43:35.484155 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:35.496958 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:35.498961 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:35.499142 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:35.503199 21056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35733
I20260809 16:43:35.503273 21132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35733 every 8 connection(s)
I20260809 16:43:35.503631 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:35.504539 21133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:35.506860 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d: Bootstrap starting.
I20260809 16:43:35.507488 21133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:35.507809 21133 log.cc:824] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:35.507930 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21056
I20260809 16:43:35.508088 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:35.508265 28922 external_mini_cluster.cc:1428] Setting key 2bca09e743e0f0b72d9ebef6023d3929
I20260809 16:43:35.508682 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d: No bootstrap required, opened a new log
I20260809 16:43:35.511435 21133 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260809 16:43:35.511615 21133 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:35.511653 21133 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d77fb5835e45dfb197f3e6de2fa54d, State: Initialized, Role: FOLLOWER
I20260809 16:43:35.511775 21133 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260809 16:43:35.511855 21133 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:35.511893 21133 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:35.511937 21133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:35.512701 21133 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260809 16:43:35.512791 21133 leader_election.cc:302] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d77fb5835e45dfb197f3e6de2fa54d; no voters:
I20260809 16:43:35.513020 21133 leader_election.cc:288] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:35.513044 21139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:35.513253 21139 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [term 1 LEADER]: Becoming Leader. State: Replica: c3d77fb5835e45dfb197f3e6de2fa54d, State: Running, Role: LEADER
I20260809 16:43:35.513391 21139 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } }
I20260809 16:43:35.513713 21133 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:35.513849 21140 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d77fb5835e45dfb197f3e6de2fa54d. Latest consensus state: current_term: 1 leader_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } } }
I20260809 16:43:35.513928 21140 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:35.514140 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d77fb5835e45dfb197f3e6de2fa54d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35733 } } }
I20260809 16:43:35.514402 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:35.514612 21143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:35.514793 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:35.508788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:34445 (local address 127.0.0.1:35733)
0809 16:43:35.508971 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:35.508978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:35.509004 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:43:35.509957 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:35.509964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:35.510013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:35.510126 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:35.510419 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:35.511348 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:35.512450 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:35.512960 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:35.513087 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:35.513851 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:35.513855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:35.513855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:35.514056 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:35.514093 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:35.514108 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:35.514190 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:35.514478 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 16:43:35.514508 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260809 16:43:35.515311 21143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:35.515952 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:35.517086 21143 catalog_manager.cc:1347] Generated new cluster ID: 178d88816461444683edf6bccdacd78d
I20260809 16:43:35.517145 21143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:35.520603 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:35.517535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:51357 (local address 127.0.0.1:35733)
0809 16:43:35.517569 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:35.517570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:35.517650 (+ 80us) server_negotiation.cc:407] Connection header received
0809 16:43:35.517709 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:35.517710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:35.517717 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:35.517759 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:35.517882 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:35.518312 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:35.519292 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:35.519575 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:35.519654 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:35.520359 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:35.520360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:35.520360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:35.520365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:35.520380 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:35.520384 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:35.520452 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:35.520524 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 16:43:35.520534 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260809 16:43:35.524253 21143 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:35.524753 21143 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:35.530191 21143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d77fb5835e45dfb197f3e6de2fa54d: Generated new TSK 0
I20260809 16:43:35.530429 21143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:35.537681 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21056
W20260809 16:43:35.545616 21040 connection.cc:535] client connection to 127.0.0.1:35733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35733): Connection reset by peer (error 104)
2026-08-09T16:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:35 dist-test-slave-wl2h krb5kdc[21169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:35 dist-test-slave-wl2h krb5kdc[21169](info): set up 2 sockets
Aug 09 16:43:35 dist-test-slave-wl2h krb5kdc[21169](info): commencing operation
krb5kdc: starting...
W20260809 16:43:36.678786 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:36 dist-test-slave-wl2h krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:36.811141 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:36.895358 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:36.895566 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:36.895589 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:36.896744 21186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:36.896788 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:36.896802 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:36.896816 21186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:36.896827 21186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:36.898432 21186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:36.898649 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:36.898844 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:36.903955 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:36.903995 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:36.904155 21186 server_base.cc:1034] running on GCE node
W20260809 16:43:36.903955 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:36.904632 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:36.904974 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:36.906219 21186 hybrid_clock.cc:648] HybridClock initialized: now 1786293816906200 us; error 32 us; skew 500 ppm
Aug 09 16:43:36 dist-test-slave-wl2h krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:36.909133 21186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:36.909634 21186 webserver.cc:460] Webserver started at http://127.0.0.1:46675/ using document root <none> and password file <none>
I20260809 16:43:36.909889 21186 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:36.909961 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:36.910115 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:36.911289 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d"
format_stamp: "Formatted at 2026-08-09 16:43:36 on dist-test-slave-wl2h"
server_key: "73d5c4447471712337d29e1d8f0814f5"
server_key_iv: "f1b3cee949e181d910778070ba8890d9"
server_key_version: "encryptionkey@0"
I20260809 16:43:36.911674 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d"
format_stamp: "Formatted at 2026-08-09 16:43:36 on dist-test-slave-wl2h"
server_key: "73d5c4447471712337d29e1d8f0814f5"
server_key_iv: "f1b3cee949e181d910778070ba8890d9"
server_key_version: "encryptionkey@0"
I20260809 16:43:36.913187 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:36.914119 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:36.914373 21186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:36.914489 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d"
format_stamp: "Formatted at 2026-08-09 16:43:36 on dist-test-slave-wl2h"
server_key: "73d5c4447471712337d29e1d8f0814f5"
server_key_iv: "f1b3cee949e181d910778070ba8890d9"
server_key_version: "encryptionkey@0"
I20260809 16:43:36.914572 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:36.918241 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:36.920558 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:36.920753 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:36.925014 21186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34605
I20260809 16:43:36.925045 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34605 every 8 connection(s)
I20260809 16:43:36.925405 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:36.926474 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:36.927073 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21186
I20260809 16:43:36.927223 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:36.927424 28922 external_mini_cluster.cc:1428] Setting key 59ffee6e5e5b5b091df8b437a5223edf
I20260809 16:43:36.930030 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d: Bootstrap starting.
I20260809 16:43:36.930254 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:36.928128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:50419 (local address 127.0.0.1:34605)
0809 16:43:36.928307 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:36.928315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:36.928340 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:43:36.928739 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:36.928745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:36.928768 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:36.928885 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:36.929608 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:36.929612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:36.929612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:36.929616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:36.929662 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:36.929672 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:36.929740 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:36.929969 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 16:43:36.930044 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260809 16:43:36.930848 21269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:36.931245 21269 log.cc:824] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:36.931417 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:36.931983 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d: No bootstrap required, opened a new log
I20260809 16:43:36.934113 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:36.932940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:33343 (local address 127.0.0.1:34605)
0809 16:43:36.932981 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:36.932983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:36.933060 (+ 77us) server_negotiation.cc:407] Connection header received
0809 16:43:36.933120 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:36.933121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:36.933126 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:36.933183 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:36.933864 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:36.933866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:36.933866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:36.933872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:36.933891 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:36.933897 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:36.933979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:36.934014 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 16:43:36.934030 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:36.934373 21269 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260809 16:43:36.934576 21269 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:36.934620 21269 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcaeed2ef0e34652b7a6ad2d9ca9822d, State: Initialized, Role: FOLLOWER
I20260809 16:43:36.934755 21269 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260809 16:43:36.934844 21269 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:36.934887 21269 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:36.934940 21269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:36.935544 21269 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260809 16:43:36.935640 21269 leader_election.cc:302] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcaeed2ef0e34652b7a6ad2d9ca9822d; no voters:
I20260809 16:43:36.935830 21269 leader_election.cc:288] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:36.935909 21282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:36.936102 21282 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [term 1 LEADER]: Becoming Leader. State: Replica: dcaeed2ef0e34652b7a6ad2d9ca9822d, State: Running, Role: LEADER
I20260809 16:43:36.936100 21269 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:36.936321 21282 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260809 16:43:36.936728 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcaeed2ef0e34652b7a6ad2d9ca9822d. Latest consensus state: current_term: 1 leader_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } } }
I20260809 16:43:36.936740 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcaeed2ef0e34652b7a6ad2d9ca9822d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } } }
I20260809 16:43:36.936839 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:36.936852 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:36.937110 21292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:36.937704 21292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:36.943631 21292 catalog_manager.cc:1347] Generated new cluster ID: 3adf95f38e4949d4a008e2531d7151c8
I20260809 16:43:36.943719 21292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:36.966967 21292 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:36.967192 21292 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:36.973960 21292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcaeed2ef0e34652b7a6ad2d9ca9822d: Generated new TSK 0
I20260809 16:43:36.974189 21292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:36.984836 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21186
2026-08-09T16:43:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:37 dist-test-slave-wl2h krb5kdc[21305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:37 dist-test-slave-wl2h krb5kdc[21305](info): set up 2 sockets
Aug 09 16:43:37 dist-test-slave-wl2h krb5kdc[21305](info): commencing operation
krb5kdc: starting...
W20260809 16:43:38.120342 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:38.250782 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:38.333000 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:38.333252 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:38.333294 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:38.334429 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:38.334503 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:38.334528 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:38.334558 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:38.334589 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:38.336261 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:38.336555 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:38.336791 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:38.342139 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:38.342159 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:38.342170 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:38.342489 21323 server_base.cc:1034] running on GCE node
I20260809 16:43:38.342746 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:38.343073 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:38.344237 21323 hybrid_clock.cc:648] HybridClock initialized: now 1786293818344218 us; error 39 us; skew 500 ppm
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:38.346903 21323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:38.347383 21323 webserver.cc:460] Webserver started at http://127.0.0.1:37775/ using document root <none> and password file <none>
I20260809 16:43:38.347628 21323 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:38.347698 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:38.347851 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:38.348944 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "7593595acffa4e10a157b1c88f7b9288"
format_stamp: "Formatted at 2026-08-09 16:43:38 on dist-test-slave-wl2h"
server_key: "6a6df1d06e414ab07d49b46e61e1d210"
server_key_iv: "07849ffe91692702b4d8511944705127"
server_key_version: "encryptionkey@0"
I20260809 16:43:38.349323 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "7593595acffa4e10a157b1c88f7b9288"
format_stamp: "Formatted at 2026-08-09 16:43:38 on dist-test-slave-wl2h"
server_key: "6a6df1d06e414ab07d49b46e61e1d210"
server_key_iv: "07849ffe91692702b4d8511944705127"
server_key_version: "encryptionkey@0"
I20260809 16:43:38.350811 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:38.351720 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:38.351913 21323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:38.352023 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "7593595acffa4e10a157b1c88f7b9288"
format_stamp: "Formatted at 2026-08-09 16:43:38 on dist-test-slave-wl2h"
server_key: "6a6df1d06e414ab07d49b46e61e1d210"
server_key_iv: "07849ffe91692702b4d8511944705127"
server_key_version: "encryptionkey@0"
I20260809 16:43:38.352110 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:38.362900 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:38.365047 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:38.365213 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:38.369294 21323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43497
I20260809 16:43:38.369341 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43497 every 8 connection(s)
I20260809 16:43:38.369709 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:38.370857 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:38.373487 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288: Bootstrap starting.
I20260809 16:43:38.374135 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:38.374446 21404 log.cc:824] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:38.375322 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288: No bootstrap required, opened a new log
I20260809 16:43:38.376303 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21323
I20260809 16:43:38.376402 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:38.376618 28922 external_mini_cluster.cc:1428] Setting key 4047dbfa446b609a57639e444bcbf83a
I20260809 16:43:38.377842 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } }
I20260809 16:43:38.378036 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:38.378149 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7593595acffa4e10a157b1c88f7b9288, State: Initialized, Role: FOLLOWER
I20260809 16:43:38.378302 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } }
I20260809 16:43:38.378417 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:38.378466 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:38.378520 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:38.379662 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } }
I20260809 16:43:38.379781 21404 leader_election.cc:302] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7593595acffa4e10a157b1c88f7b9288; no voters:
I20260809 16:43:38.379947 21404 leader_election.cc:288] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:38.380010 21409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:38.380204 21409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [term 1 LEADER]: Becoming Leader. State: Replica: 7593595acffa4e10a157b1c88f7b9288, State: Running, Role: LEADER
I20260809 16:43:38.380213 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:38.380364 21409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } }
I20260809 16:43:38.380832 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7593595acffa4e10a157b1c88f7b9288. Latest consensus state: current_term: 1 leader_uuid: "7593595acffa4e10a157b1c88f7b9288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } } }
I20260809 16:43:38.381100 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:38.381008 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7593595acffa4e10a157b1c88f7b9288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593595acffa4e10a157b1c88f7b9288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43497 } } }
I20260809 16:43:38.381716 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:38.381915 21424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:38.382511 21424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:38.384348 21424 catalog_manager.cc:1347] Generated new cluster ID: 5da7a1a96adc4fcd81488264d8553d2f
I20260809 16:43:38.384406 21424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 16:43:38.389609 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:38.377160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:35551 (local address 127.0.0.1:43497)
0809 16:43:38.377435 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:38.377445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:38.377483 (+ 38us) server_negotiation.cc:407] Connection header received
0809 16:43:38.377797 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:38.377806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:38.377875 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:38.378031 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:38.378886 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:38.380405 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:38.381680 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:38.382378 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:38.382513 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:38.385538 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:38.385542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:38.385543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:38.385555 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:38.387694 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:38.388189 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:38.388192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:38.388192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:38.388334 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:38.388586 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:38.388587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:38.388587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:38.388914 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:38.389089 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:38.389203 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 16:43:38.389339 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":130,"threads_started":1}
I20260809 16:43:38.390687 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:38.397454 21424 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:38.398257 21424 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:38.400144 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:38.392320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38465 (local address 127.0.0.1:43497)
0809 16:43:38.392360 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:38.392362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:38.392500 (+ 138us) server_negotiation.cc:407] Connection header received
0809 16:43:38.392547 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:38.392548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:38.392558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:38.392621 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:38.393255 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:38.394036 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:38.395156 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:38.395489 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:38.395566 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:38.397074 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:38.397075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:38.397075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:38.397082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:38.398827 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:38.399292 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:38.399293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:38.399294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:38.399355 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:38.399605 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:38.399605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:38.399606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:38.399744 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:38.399842 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:38.399977 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 16:43:38.400041 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:38.413666 21424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7593595acffa4e10a157b1c88f7b9288: Generated new TSK 0
I20260809 16:43:38.413867 21424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:38.422032 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21323
W20260809 16:43:38.432257 21303 connection.cc:535] client connection to 127.0.0.1:43497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43497): Connection reset by peer (error 104)
2026-08-09T16:43:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21437](info): set up 2 sockets
Aug 09 16:43:38 dist-test-slave-wl2h krb5kdc[21437](info): commencing operation
krb5kdc: starting...
W20260809 16:43:39.554875 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:39 dist-test-slave-wl2h krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:39.686653 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:39.767794 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:39.768044 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:39.768083 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:39.769217 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:39.769292 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:39.769317 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:39.769344 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:39.769375 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:39.770848 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:39.771152 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:39.771384 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:39.776435 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:39.776494 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:39.776510 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:39.776672 21450 server_base.cc:1034] running on GCE node
I20260809 16:43:39.776913 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:39.777173 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:39.778326 21450 hybrid_clock.cc:648] HybridClock initialized: now 1786293819778304 us; error 39 us; skew 500 ppm
Aug 09 16:43:39 dist-test-slave-wl2h krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:39.780881 21450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:39.781458 21450 webserver.cc:460] Webserver started at http://127.0.0.1:44567/ using document root <none> and password file <none>
I20260809 16:43:39.781689 21450 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:39.781735 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:39.781878 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:39.782974 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "a2d1b9bd169c4cceb7a31442cb4a183f"
format_stamp: "Formatted at 2026-08-09 16:43:39 on dist-test-slave-wl2h"
server_key: "6065b13b644587bad809c0a7443180bb"
server_key_iv: "d3168409670140adff0de1faea3f2606"
server_key_version: "encryptionkey@0"
I20260809 16:43:39.783402 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "a2d1b9bd169c4cceb7a31442cb4a183f"
format_stamp: "Formatted at 2026-08-09 16:43:39 on dist-test-slave-wl2h"
server_key: "6065b13b644587bad809c0a7443180bb"
server_key_iv: "d3168409670140adff0de1faea3f2606"
server_key_version: "encryptionkey@0"
I20260809 16:43:39.784775 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:39.785655 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:39.785938 21450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:39.786019 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "a2d1b9bd169c4cceb7a31442cb4a183f"
format_stamp: "Formatted at 2026-08-09 16:43:39 on dist-test-slave-wl2h"
server_key: "6065b13b644587bad809c0a7443180bb"
server_key_iv: "d3168409670140adff0de1faea3f2606"
server_key_version: "encryptionkey@0"
I20260809 16:43:39.786108 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:39.814990 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:39.817377 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:39.817529 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:39.821995 21450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37345
I20260809 16:43:39.822077 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37345 every 8 connection(s)
I20260809 16:43:39.822513 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:39.823503 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:39.825788 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f: Bootstrap starting.
I20260809 16:43:39.826403 21532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:39.826699 21532 log.cc:824] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:39.827498 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f: No bootstrap required, opened a new log
I20260809 16:43:39.829139 21532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260809 16:43:39.829279 21532 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:39.829303 21532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d1b9bd169c4cceb7a31442cb4a183f, State: Initialized, Role: FOLLOWER
I20260809 16:43:39.829454 21532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260809 16:43:39.829516 21532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:39.829557 21532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:39.829608 21532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:39.830173 21532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260809 16:43:39.830282 21532 leader_election.cc:302] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d1b9bd169c4cceb7a31442cb4a183f; no voters:
I20260809 16:43:39.830483 21532 leader_election.cc:288] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:39.830531 21536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:39.830740 21536 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [term 1 LEADER]: Becoming Leader. State: Replica: a2d1b9bd169c4cceb7a31442cb4a183f, State: Running, Role: LEADER
I20260809 16:43:39.830749 21532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:39.830847 21536 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260809 16:43:39.831285 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d1b9bd169c4cceb7a31442cb4a183f. Latest consensus state: current_term: 1 leader_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } } }
I20260809 16:43:39.831442 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:39.831435 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d1b9bd169c4cceb7a31442cb4a183f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } } }
I20260809 16:43:39.831938 21545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:39.832001 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:39.832098 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21450
I20260809 16:43:39.832252 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:39.832454 28922 external_mini_cluster.cc:1428] Setting key 4a4f9b114e6fad90f223ea8d6e1baa91
I20260809 16:43:39.832494 21545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:39.834613 21545 catalog_manager.cc:1347] Generated new cluster ID: 36c48532046940a48ba7322fdf5366cc
I20260809 16:43:39.834681 21545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:39.839056 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:39.832975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:60615 (local address 127.0.0.1:37345)
0809 16:43:39.833288 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:39.833299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:39.833737 (+ 438us) server_negotiation.cc:407] Connection header received
0809 16:43:39.833866 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:39.833873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:39.833933 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:39.834098 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:39.834296 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:39.835676 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:39.836817 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:39.837418 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:39.837515 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:39.838310 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:39.838313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:39.838314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:39.838533 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:39.838578 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:39.838591 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:39.838668 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:39.838800 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 16:43:39.838826 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":77,"threads_started":1}
I20260809 16:43:39.840003 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:39.843112 21545 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:39.843823 21545 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:39.845098 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:39.841623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38693 (local address 127.0.0.1:37345)
0809 16:43:39.841663 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:39.841665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:39.841681 (+ 16us) server_negotiation.cc:407] Connection header received
0809 16:43:39.841746 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:39.841746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:39.841760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:39.841832 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:39.841946 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:39.842719 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:39.843730 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:39.844049 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:39.844131 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:39.844816 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:39.844817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:39.844818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:39.844822 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:39.844837 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:39.844841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:39.844908 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:39.845002 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:43:39.845014 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:39.848243 21545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d1b9bd169c4cceb7a31442cb4a183f: Generated new TSK 0
I20260809 16:43:39.848403 21545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:39.850821 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21450
W20260809 16:43:39.858166 21435 connection.cc:535] client connection to 127.0.0.1:37345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37345): Connection reset by peer (error 104)
2026-08-09T16:43:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:39 dist-test-slave-wl2h krb5kdc[21565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:39 dist-test-slave-wl2h krb5kdc[21565](info): set up 2 sockets
Aug 09 16:43:39 dist-test-slave-wl2h krb5kdc[21565](info): commencing operation
krb5kdc: starting...
W20260809 16:43:41.028695 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:41 dist-test-slave-wl2h krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 16:43:41.154124 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:41.237640 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:41.237864 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:41.237890 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:41.238966 21581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:41.239004 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:41.239092 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:41.239113 21581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:41.239133 21581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:41.240646 21581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:41.240904 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:41.241145 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:41.246140 21588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:41.246212 21581 server_base.cc:1034] running on GCE node
W20260809 16:43:41.246119 21587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:41.246097 21590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:41.246606 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:41.246906 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:41.248049 21581 hybrid_clock.cc:648] HybridClock initialized: now 1786293821248029 us; error 38 us; skew 500 ppm
Aug 09 16:43:41 dist-test-slave-wl2h krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:41.250617 21581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 16:43:41.251185 21581 webserver.cc:460] Webserver started at http://127.0.0.1:36775/ using document root <none> and password file <none>
I20260809 16:43:41.251431 21581 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:41.251505 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:41.251646 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:41.252737 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "fee42ce8457f411f878f95cec1a0f0a4"
format_stamp: "Formatted at 2026-08-09 16:43:41 on dist-test-slave-wl2h"
server_key: "be53f14be3b6cb3789b0c13d9b43cffe"
server_key_iv: "9a55618b37755e21f52797668b50a380"
server_key_version: "encryptionkey@0"
I20260809 16:43:41.253127 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "fee42ce8457f411f878f95cec1a0f0a4"
format_stamp: "Formatted at 2026-08-09 16:43:41 on dist-test-slave-wl2h"
server_key: "be53f14be3b6cb3789b0c13d9b43cffe"
server_key_iv: "9a55618b37755e21f52797668b50a380"
server_key_version: "encryptionkey@0"
I20260809 16:43:41.254496 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:41.255358 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:41.255538 21581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:41.255646 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "fee42ce8457f411f878f95cec1a0f0a4"
format_stamp: "Formatted at 2026-08-09 16:43:41 on dist-test-slave-wl2h"
server_key: "be53f14be3b6cb3789b0c13d9b43cffe"
server_key_iv: "9a55618b37755e21f52797668b50a380"
server_key_version: "encryptionkey@0"
I20260809 16:43:41.255721 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:41.259064 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:41.261087 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:41.261256 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:41.265552 21581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42211
I20260809 16:43:41.265612 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42211 every 8 connection(s)
I20260809 16:43:41.265959 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:41.266935 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:41.269169 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4: Bootstrap starting.
I20260809 16:43:41.269687 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21581
I20260809 16:43:41.269765 21661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:41.269817 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:41.269984 28922 external_mini_cluster.cc:1428] Setting key 9479db61c99ce11da39aeb17b169e5d4
I20260809 16:43:41.270099 21661 log.cc:824] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:41.270931 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4: No bootstrap required, opened a new log
I20260809 16:43:41.272612 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:41.270538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:34009 (local address 127.0.0.1:42211)
0809 16:43:41.270731 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:41.270741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:41.270773 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:43:41.271303 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:41.271309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:41.271334 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:41.271483 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:41.272150 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:41.272154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:41.272154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:41.272158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:41.272209 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:41.272219 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:41.272347 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:41.272401 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 16:43:41.272444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260809 16:43:41.273509 21661 raft_consensus.cc:348] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } }
I20260809 16:43:41.273685 21661 raft_consensus.cc:374] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:41.273697 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:41.273736 21661 raft_consensus.cc:729] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee42ce8457f411f878f95cec1a0f0a4, State: Initialized, Role: FOLLOWER
I20260809 16:43:41.273859 21661 consensus_queue.cc:260] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } }
I20260809 16:43:41.273965 21661 raft_consensus.cc:388] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:41.274012 21661 raft_consensus.cc:482] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:41.274061 21661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:41.274606 21661 raft_consensus.cc:504] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } }
I20260809 16:43:41.274742 21661 leader_election.cc:302] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fee42ce8457f411f878f95cec1a0f0a4; no voters:
I20260809 16:43:41.274899 21661 leader_election.cc:288] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:41.274921 21670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:41.275132 21670 raft_consensus.cc:686] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [term 1 LEADER]: Becoming Leader. State: Replica: fee42ce8457f411f878f95cec1a0f0a4, State: Running, Role: LEADER
I20260809 16:43:41.275235 21661 sys_catalog.cc:564] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:41.275359 21670 consensus_queue.cc:237] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } }
I20260809 16:43:41.275692 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fee42ce8457f411f878f95cec1a0f0a4. Latest consensus state: current_term: 1 leader_uuid: "fee42ce8457f411f878f95cec1a0f0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } } }
I20260809 16:43:41.275728 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fee42ce8457f411f878f95cec1a0f0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee42ce8457f411f878f95cec1a0f0a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42211 } } }
I20260809 16:43:41.275839 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:41.275784 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:41.276083 21683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:41.276643 21683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:41.277077 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:41.275175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:50987 (local address 127.0.0.1:42211)
0809 16:43:41.275211 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:41.275213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:41.275519 (+ 306us) server_negotiation.cc:407] Connection header received
0809 16:43:41.275676 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:41.275677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:41.275681 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:41.275713 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:41.276788 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:41.276789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:41.276790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:41.276795 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:41.276809 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:41.276813 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:41.276876 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:41.276976 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 16:43:41.276994 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:41.278647 21683 catalog_manager.cc:1347] Generated new cluster ID: 703c2df46124465ba259ab9d5d60bf60
I20260809 16:43:41.278704 21683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:41.288308 21683 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:41.288509 21683 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:41.291183 21683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fee42ce8457f411f878f95cec1a0f0a4: Generated new TSK 0
I20260809 16:43:41.291361 21683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:41.303681 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21581
2026-08-09T16:43:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:41 dist-test-slave-wl2h krb5kdc[21700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:41 dist-test-slave-wl2h krb5kdc[21700](info): set up 2 sockets
Aug 09 16:43:41 dist-test-slave-wl2h krb5kdc[21700](info): commencing operation
krb5kdc: starting...
W20260809 16:43:42.442570 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:42.569434 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44233
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:42.651196 21713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:42.651402 21713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:42.651425 21713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:42.652501 21713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:42.652539 21713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:42.652554 21713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:42.652566 21713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:42.652578 21713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:42.653965 21713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:42.654147 21713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:42.654313 21713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:42.659204 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:42.659214 21718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:42.659255 21719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:42.659572 21713 server_base.cc:1034] running on GCE node
I20260809 16:43:42.659873 21713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:42.660203 21713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:42.661407 21713 hybrid_clock.cc:648] HybridClock initialized: now 1786293822661394 us; error 50 us; skew 500 ppm
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:42.663970 21713 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:42.664358 21713 webserver.cc:460] Webserver started at http://127.28.62.190:36483/ using document root <none> and password file <none>
I20260809 16:43:42.664605 21713 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:42.664650 21713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:42.664738 21713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:42.665722 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "4de6da0ed9c648f388bf2e1a344a9554"
format_stamp: "Formatted at 2026-08-09 16:43:42 on dist-test-slave-wl2h"
server_key: "4b3577e07fd03446953196848e033cea"
server_key_iv: "4063b9a11a7b4d8848a2407c604866e5"
server_key_version: "encryptionkey@0"
I20260809 16:43:42.666028 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "4de6da0ed9c648f388bf2e1a344a9554"
format_stamp: "Formatted at 2026-08-09 16:43:42 on dist-test-slave-wl2h"
server_key: "4b3577e07fd03446953196848e033cea"
server_key_iv: "4063b9a11a7b4d8848a2407c604866e5"
server_key_version: "encryptionkey@0"
I20260809 16:43:42.667403 21713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:42.668105 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:42.668318 21713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:42.668408 21713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "4de6da0ed9c648f388bf2e1a344a9554"
format_stamp: "Formatted at 2026-08-09 16:43:42 on dist-test-slave-wl2h"
server_key: "4b3577e07fd03446953196848e033cea"
server_key_iv: "4063b9a11a7b4d8848a2407c604866e5"
server_key_version: "encryptionkey@0"
I20260809 16:43:42.668458 21713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:42.683732 21713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:42.685863 21713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:42.685979 21713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:42.690204 21713 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44233
I20260809 16:43:42.690269 21789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44233 every 8 connection(s)
I20260809 16:43:42.690615 21713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:42.691582 21790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:42.693981 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554: Bootstrap starting.
I20260809 16:43:42.694581 21790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:42.694950 21790 log.cc:824] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:42.695045 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21713
I20260809 16:43:42.695160 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:42.695372 28922 external_mini_cluster.cc:1428] Setting key 611f5dca55fa1e6cbf1bbcaea42916c0
I20260809 16:43:42.695808 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554: No bootstrap required, opened a new log
I20260809 16:43:42.697544 21790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } }
I20260809 16:43:42.697712 21790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:42.697777 21790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de6da0ed9c648f388bf2e1a344a9554, State: Initialized, Role: FOLLOWER
I20260809 16:43:42.697957 21790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } }
I20260809 16:43:42.698053 21790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:42.698104 21790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:42.698153 21790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:42.699079 21790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } }
I20260809 16:43:42.699213 21790 leader_election.cc:302] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de6da0ed9c648f388bf2e1a344a9554; no voters:
I20260809 16:43:42.699409 21790 leader_election.cc:288] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:42.699431 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:42.699551 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [term 1 LEADER]: Becoming Leader. State: Replica: 4de6da0ed9c648f388bf2e1a344a9554, State: Running, Role: LEADER
I20260809 16:43:42.699678 21790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:42.699697 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } }
I20260809 16:43:42.700098 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de6da0ed9c648f388bf2e1a344a9554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } } }
I20260809 16:43:42.700184 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:42.700448 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de6da0ed9c648f388bf2e1a344a9554. Latest consensus state: current_term: 1 leader_uuid: "4de6da0ed9c648f388bf2e1a344a9554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de6da0ed9c648f388bf2e1a344a9554" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44233 } } }
I20260809 16:43:42.700572 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:42.700760 21802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:42.701313 21802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:42.704310 21802 catalog_manager.cc:1347] Generated new cluster ID: b05485ba09414c40820d8425750ea98e
I20260809 16:43:42.704367 21802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:42.708290 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:42.695935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41425 (local address 127.28.62.190:44233)
0809 16:43:42.696149 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:42.696159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:42.696195 (+ 36us) server_negotiation.cc:407] Connection header received
0809 16:43:42.697531 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:42.697537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:42.697590 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:42.697738 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:42.698498 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:42.699671 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:42.700775 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:42.701390 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:42.701823 (+ 433us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:42.704516 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:42.704520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:42.704520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:42.704530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:42.706315 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:42.706717 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:42.706720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:42.706720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:42.706836 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:42.707097 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:42.707098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:42.707098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:42.707335 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:42.707997 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:42.708066 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 16:43:42.708146 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260809 16:43:42.709249 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:42.721607 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:42.715328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44961 (local address 127.28.62.190:44233)
0809 16:43:42.715380 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:42.715381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:42.715548 (+ 167us) server_negotiation.cc:407] Connection header received
0809 16:43:42.715621 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:42.715622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:42.715632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:42.715693 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:42.716225 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:42.716743 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:42.717441 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:42.717662 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:42.717749 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:42.718912 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:42.718914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:42.718914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:42.718921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:42.720413 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:42.720790 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:42.720791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:42.720791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:42.720854 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:42.721089 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:42.721090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:42.721090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:42.721205 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:42.721315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:42.721441 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 16:43:42.721509 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 16:43:42.746781 21802 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:42.747421 21802 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:42.755609 21802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de6da0ed9c648f388bf2e1a344a9554: Generated new TSK 0
I20260809 16:43:42.755832 21802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:42.761096 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21713
W20260809 16:43:42.768712 21697 connection.cc:535] client connection to 127.28.62.190:44233 recv error: Network error: failed to read from TLS socket (remote: 127.28.62.190:44233): Connection reset by peer (error 104)
2026-08-09T16:43:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21825](info): set up 2 sockets
Aug 09 16:43:42 dist-test-slave-wl2h krb5kdc[21825](info): commencing operation
krb5kdc: starting...
W20260809 16:43:43.891901 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:43 dist-test-slave-wl2h krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:44.022921 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35097
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:44.109827 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:44.110039 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:44.110064 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:44.111279 21844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:44.111320 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:44.111335 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:44.111348 21844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:44.111361 21844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:44.112890 21844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:35097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:44.113152 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:44.113437 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:44.118395 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:44.118373 21849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:44.118546 21844 server_base.cc:1034] running on GCE node
W20260809 16:43:44.118383 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:44.118837 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:44.119149 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:44.120330 21844 hybrid_clock.cc:648] HybridClock initialized: now 1786293824120319 us; error 32 us; skew 500 ppm
Aug 09 16:43:44 dist-test-slave-wl2h krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:44.123131 21844 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:44.123565 21844 webserver.cc:460] Webserver started at http://127.28.62.190:40485/ using document root <none> and password file <none>
I20260809 16:43:44.123798 21844 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:44.123843 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:44.124008 21844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:44.125082 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "b8fb9fa9bf324825951c76bb4add8374"
format_stamp: "Formatted at 2026-08-09 16:43:44 on dist-test-slave-wl2h"
server_key: "2d88c8b28f98c1598d1a5426edc0fa30"
server_key_iv: "377782997dce062955de05f599d52945"
server_key_version: "encryptionkey@0"
I20260809 16:43:44.125449 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "b8fb9fa9bf324825951c76bb4add8374"
format_stamp: "Formatted at 2026-08-09 16:43:44 on dist-test-slave-wl2h"
server_key: "2d88c8b28f98c1598d1a5426edc0fa30"
server_key_iv: "377782997dce062955de05f599d52945"
server_key_version: "encryptionkey@0"
I20260809 16:43:44.126789 21844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:44.127674 21861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:44.127880 21844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:44.127986 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "b8fb9fa9bf324825951c76bb4add8374"
format_stamp: "Formatted at 2026-08-09 16:43:44 on dist-test-slave-wl2h"
server_key: "2d88c8b28f98c1598d1a5426edc0fa30"
server_key_iv: "377782997dce062955de05f599d52945"
server_key_version: "encryptionkey@0"
I20260809 16:43:44.128130 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:44.141865 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:44.144260 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:44.144436 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:44.148783 21844 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:35097
I20260809 16:43:44.148818 21927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:35097 every 8 connection(s)
I20260809 16:43:44.149179 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:44.149935 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21844
I20260809 16:43:44.150071 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:44.150166 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:44.150287 28922 external_mini_cluster.cc:1428] Setting key 07a2e298a5b2eb73a7307e0cc7ead01a
I20260809 16:43:44.153606 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374: Bootstrap starting.
I20260809 16:43:44.154337 21928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:44.154644 21928 log.cc:824] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:44.155447 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374: No bootstrap required, opened a new log
I20260809 16:43:44.156661 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:44.150846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.28.62.190:35097)
0809 16:43:44.151100 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:44.151109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:44.151136 (+ 27us) server_negotiation.cc:407] Connection header received
0809 16:43:44.151742 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:44.151749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:44.151794 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:44.151907 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:44.152174 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:44.153051 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:44.154235 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:44.154836 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:44.154952 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:44.155862 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:44.155866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:44.155866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:44.156056 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:44.156085 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:44.156096 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:44.156197 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:44.156278 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 16:43:44.156307 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260809 16:43:44.156903 21928 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } }
I20260809 16:43:44.157080 21928 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:44.157119 21928 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fb9fa9bf324825951c76bb4add8374, State: Initialized, Role: FOLLOWER
I20260809 16:43:44.157258 21928 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } }
I20260809 16:43:44.157393 21928 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:44.157446 21928 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:44.157501 21928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:44.157732 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:44.158212 21928 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } }
I20260809 16:43:44.158337 21928 leader_election.cc:302] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fb9fa9bf324825951c76bb4add8374; no voters:
I20260809 16:43:44.158502 21928 leader_election.cc:288] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:44.158573 21934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:44.158756 21928 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:44.158767 21934 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [term 1 LEADER]: Becoming Leader. State: Replica: b8fb9fa9bf324825951c76bb4add8374, State: Running, Role: LEADER
I20260809 16:43:44.158964 21934 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } }
I20260809 16:43:44.159377 21935 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8fb9fa9bf324825951c76bb4add8374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } } }
I20260809 16:43:44.159456 21935 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:44.159632 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fb9fa9bf324825951c76bb4add8374. Latest consensus state: current_term: 1 leader_uuid: "b8fb9fa9bf324825951c76bb4add8374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb9fa9bf324825951c76bb4add8374" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 35097 } } }
I20260809 16:43:44.159688 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:44.160113 21946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:44.160660 21946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:44.162182 21946 catalog_manager.cc:1347] Generated new cluster ID: 1795da82942343649e41bfad65649609
I20260809 16:43:44.162241 21946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:44.163323 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:44.159349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52749 (local address 127.28.62.190:35097)
0809 16:43:44.159386 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:44.159388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:44.159685 (+ 297us) server_negotiation.cc:407] Connection header received
0809 16:43:44.159812 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:44.159813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:44.159825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:44.159894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:44.160065 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:44.160864 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:44.161925 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:44.162220 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:44.162291 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:44.163084 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:44.163085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:44.163085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:44.163090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:44.163102 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:44.163106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:44.163135 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:44.163245 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 16:43:44.163254 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 16:43:44.183589 21946 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:44.184211 21946 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:44.191021 21946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8fb9fa9bf324825951c76bb4add8374: Generated new TSK 0
I20260809 16:43:44.191229 21946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:44.202090 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21844
W20260809 16:43:44.211337 21823 connection.cc:535] client connection to 127.28.62.190:35097 recv error: Network error: failed to read from TLS socket (remote: 127.28.62.190:35097): Connection reset by peer (error 104)
2026-08-09T16:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:44 dist-test-slave-wl2h krb5kdc[21966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:44 dist-test-slave-wl2h krb5kdc[21966](info): set up 2 sockets
Aug 09 16:43:44 dist-test-slave-wl2h krb5kdc[21966](info): commencing operation
krb5kdc: starting...
W20260809 16:43:45.407506 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:45 dist-test-slave-wl2h krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:45.538936 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42469
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:45.623044 21982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:45.623313 21982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:45.623356 21982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:45.625133 21982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:45.625208 21982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:45.625238 21982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:45.625262 21982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:45.625293 21982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:45.628037 21982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:42469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:45.628360 21982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:45.628712 21982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:45.634891 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:45.634902 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:45.635159 21982 server_base.cc:1034] running on GCE node
W20260809 16:43:45.634894 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:45.635551 21982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:45.635819 21982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:45.636965 21982 hybrid_clock.cc:648] HybridClock initialized: now 1786293825636951 us; error 31 us; skew 500 ppm
Aug 09 16:43:45 dist-test-slave-wl2h krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:45.643237 21982 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:45.643796 21982 webserver.cc:460] Webserver started at http://127.28.62.190:38479/ using document root <none> and password file <none>
I20260809 16:43:45.643996 21982 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:45.644042 21982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:45.644145 21982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:45.645196 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "3efc539c1270465ca86ab2dcbd4c0726"
format_stamp: "Formatted at 2026-08-09 16:43:45 on dist-test-slave-wl2h"
server_key: "ec6ee3ff988f80043f37e6fe791728ec"
server_key_iv: "56a379165273effad3bab0336b180a67"
server_key_version: "encryptionkey@0"
I20260809 16:43:45.645510 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "3efc539c1270465ca86ab2dcbd4c0726"
format_stamp: "Formatted at 2026-08-09 16:43:45 on dist-test-slave-wl2h"
server_key: "ec6ee3ff988f80043f37e6fe791728ec"
server_key_iv: "56a379165273effad3bab0336b180a67"
server_key_version: "encryptionkey@0"
I20260809 16:43:45.646905 21982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:45.647840 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:45.648092 21982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:45.648176 21982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "3efc539c1270465ca86ab2dcbd4c0726"
format_stamp: "Formatted at 2026-08-09 16:43:45 on dist-test-slave-wl2h"
server_key: "ec6ee3ff988f80043f37e6fe791728ec"
server_key_iv: "56a379165273effad3bab0336b180a67"
server_key_version: "encryptionkey@0"
I20260809 16:43:45.648274 21982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:45.651841 21982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:45.654253 21982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:45.654417 21982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:45.658672 21982 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:42469
I20260809 16:43:45.658710 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:42469 every 8 connection(s)
I20260809 16:43:45.659118 21982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:45.660063 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:45.662352 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726: Bootstrap starting.
I20260809 16:43:45.662936 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:45.663270 22064 log.cc:824] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:45.663983 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726: No bootstrap required, opened a new log
I20260809 16:43:45.664815 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 21982
I20260809 16:43:45.664932 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:45.665114 28922 external_mini_cluster.cc:1428] Setting key c644c9d5b2a5aa2e151dccd4533d02c6
I20260809 16:43:45.666236 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } }
I20260809 16:43:45.666395 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:45.666460 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efc539c1270465ca86ab2dcbd4c0726, State: Initialized, Role: FOLLOWER
I20260809 16:43:45.666617 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } }
I20260809 16:43:45.666718 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:45.666762 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:45.666811 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:45.667795 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:45.665643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53979 (local address 127.28.62.190:42469)
0809 16:43:45.665828 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:45.665836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:45.665866 (+ 30us) server_negotiation.cc:407] Connection header received
0809 16:43:45.666239 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:45.666247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:45.666279 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:45.666465 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:45.667373 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:45.667376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:45.667377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:45.667381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:45.667432 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:45.667442 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:45.667521 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:45.667600 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 16:43:45.667642 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260809 16:43:45.667817 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } }
I20260809 16:43:45.667948 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efc539c1270465ca86ab2dcbd4c0726; no voters:
I20260809 16:43:45.668119 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:45.668174 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:45.668345 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [term 1 LEADER]: Becoming Leader. State: Replica: 3efc539c1270465ca86ab2dcbd4c0726, State: Running, Role: LEADER
I20260809 16:43:45.668397 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:45.668468 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } }
I20260809 16:43:45.668779 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:45.668839 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efc539c1270465ca86ab2dcbd4c0726. Latest consensus state: current_term: 1 leader_uuid: "3efc539c1270465ca86ab2dcbd4c0726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } } }
I20260809 16:43:45.668850 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3efc539c1270465ca86ab2dcbd4c0726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efc539c1270465ca86ab2dcbd4c0726" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 42469 } } }
I20260809 16:43:45.668936 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:45.668939 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:45.669708 22081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:45.670234 22081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:45.671912 22081 catalog_manager.cc:1347] Generated new cluster ID: 1b1e280ccfd943f19802092683ae97c6
I20260809 16:43:45.671970 22081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:45.672448 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:45.670543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33085 (local address 127.28.62.190:42469)
0809 16:43:45.670584 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:45.670587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:45.671202 (+ 615us) server_negotiation.cc:407] Connection header received
0809 16:43:45.671255 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:45.671256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:45.671261 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:45.671322 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:45.672201 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:45.672203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:45.672203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:45.672207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:45.672222 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:45.672226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:45.672258 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:45.672367 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 16:43:45.672381 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:45.690358 22081 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:45.690584 22081 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:45.694419 22081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3efc539c1270465ca86ab2dcbd4c0726: Generated new TSK 0
I20260809 16:43:45.694609 22081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:45.696477 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 21982
2026-08-09T16:43:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:45 dist-test-slave-wl2h krb5kdc[22103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:45 dist-test-slave-wl2h krb5kdc[22103](info): set up 2 sockets
Aug 09 16:43:45 dist-test-slave-wl2h krb5kdc[22103](info): commencing operation
krb5kdc: starting...
W20260809 16:43:46.828161 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:46 dist-test-slave-wl2h krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:46.959242 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36851
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:36851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:47.042177 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:47.042418 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:47.042457 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:47.043613 22120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:47.043699 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:47.043725 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:47.043756 22120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:47.043778 22120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:47.045213 22120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:36851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:36851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:47.045492 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:47.045728 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:47.050557 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:47.050539 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:47.050539 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:47.050912 22120 server_base.cc:1034] running on GCE node
I20260809 16:43:47.051187 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:47.051491 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:47.052675 22120 hybrid_clock.cc:648] HybridClock initialized: now 1786293827052659 us; error 49 us; skew 500 ppm
Aug 09 16:43:47 dist-test-slave-wl2h krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293827, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:47.055276 22120 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:47.055747 22120 webserver.cc:460] Webserver started at http://127.28.62.190:37163/ using document root <none> and password file <none>
I20260809 16:43:47.055930 22120 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:47.055969 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:47.056054 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:47.057063 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "625338d910c4445ea6845dc88812a84b"
format_stamp: "Formatted at 2026-08-09 16:43:47 on dist-test-slave-wl2h"
server_key: "b6886bfb317170d284d61d1b79319aac"
server_key_iv: "e83667b2d1ffc69e5d762c08c95d58ef"
server_key_version: "encryptionkey@0"
I20260809 16:43:47.057423 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "625338d910c4445ea6845dc88812a84b"
format_stamp: "Formatted at 2026-08-09 16:43:47 on dist-test-slave-wl2h"
server_key: "b6886bfb317170d284d61d1b79319aac"
server_key_iv: "e83667b2d1ffc69e5d762c08c95d58ef"
server_key_version: "encryptionkey@0"
I20260809 16:43:47.058738 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260809 16:43:47.059522 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:47.059713 22120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:47.059806 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "625338d910c4445ea6845dc88812a84b"
format_stamp: "Formatted at 2026-08-09 16:43:47 on dist-test-slave-wl2h"
server_key: "b6886bfb317170d284d61d1b79319aac"
server_key_iv: "e83667b2d1ffc69e5d762c08c95d58ef"
server_key_version: "encryptionkey@0"
I20260809 16:43:47.059864 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:47.073829 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:47.076210 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:47.076328 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:47.080464 22120 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:36851
I20260809 16:43:47.080500 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:36851 every 8 connection(s)
I20260809 16:43:47.080865 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:47.081765 22196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:47.084178 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: Bootstrap starting.
I20260809 16:43:47.084813 22196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:47.084913 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22120
I20260809 16:43:47.085039 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:47.085206 22196 log.cc:824] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:47.085271 28922 external_mini_cluster.cc:1428] Setting key 9ca241d11b5b5af8aefc3731531bb086
I20260809 16:43:47.086084 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: No bootstrap required, opened a new log
I20260809 16:43:47.088464 22196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } }
I20260809 16:43:47.088626 22196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:47.088678 22196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625338d910c4445ea6845dc88812a84b, State: Initialized, Role: FOLLOWER
I20260809 16:43:47.088822 22196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } }
I20260809 16:43:47.088922 22196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:47.088968 22196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:47.089017 22196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:47.089591 22196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } }
I20260809 16:43:47.089709 22196 leader_election.cc:302] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625338d910c4445ea6845dc88812a84b; no voters:
I20260809 16:43:47.089926 22196 leader_election.cc:288] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:47.089949 22201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:47.090132 22201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [term 1 LEADER]: Becoming Leader. State: Replica: 625338d910c4445ea6845dc88812a84b, State: Running, Role: LEADER
I20260809 16:43:47.090250 22196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:47.090279 22201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } }
I20260809 16:43:47.090793 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625338d910c4445ea6845dc88812a84b. Latest consensus state: current_term: 1 leader_uuid: "625338d910c4445ea6845dc88812a84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } } }
I20260809 16:43:47.091073 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:47.090971 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625338d910c4445ea6845dc88812a84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625338d910c4445ea6845dc88812a84b" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 36851 } } }
I20260809 16:43:47.091339 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b [sys.catalog]: This master's current role is: LEADER
W20260809 16:43:47.092271 22221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:47.092376 22221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:47.092505 22210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 09 16:43:47 dist-test-slave-wl2h krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:47.092742 22210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:47.095569 22210 catalog_manager.cc:1347] Generated new cluster ID: 0ede08d5edf54361b88d934a39528d17
I20260809 16:43:47.095633 22210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:47.096958 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:47.085815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39977 (local address 127.28.62.190:36851)
0809 16:43:47.085990 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:47.086001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:47.086038 (+ 37us) server_negotiation.cc:407] Connection header received
0809 16:43:47.086588 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:47.086595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:47.086651 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:47.086815 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:47.087542 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:47.088443 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:47.089392 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:47.089911 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:47.090026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:47.093149 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:47.093153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:47.093154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:47.093166 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:47.095266 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:47.095755 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:47.095759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:47.095759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:47.095876 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:47.096114 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:47.096115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:47.096116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:47.096364 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:47.096550 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:47.096643 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 16:43:47.096786 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260809 16:43:47.100127 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:47.110426 22210 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:47.111214 22210 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:47.113677 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:47.101586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42789 (local address 127.28.62.190:36851)
0809 16:43:47.101626 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:47.101628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:47.106120 (+ 4492us) server_negotiation.cc:407] Connection header received
0809 16:43:47.106147 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:47.106148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:47.106161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:47.106240 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:47.107361 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:47.108414 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:47.109442 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:47.109680 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:47.109770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:47.111052 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:47.111053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:47.111054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:47.111062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:47.112529 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:47.112891 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:47.112892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:47.112892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:47.112950 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:47.113159 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:47.113160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:47.113161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:47.113268 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:47.113365 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:47.113483 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 16:43:47.113571 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:47.115864 22210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625338d910c4445ea6845dc88812a84b: Generated new TSK 0
I20260809 16:43:47.116031 22210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:47.118180 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22120
2026-08-09T16:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:47 dist-test-slave-wl2h krb5kdc[22234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:47 dist-test-slave-wl2h krb5kdc[22234](info): set up 2 sockets
Aug 09 16:43:47 dist-test-slave-wl2h krb5kdc[22234](info): commencing operation
krb5kdc: starting...
W20260809 16:43:48.257347 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:48 dist-test-slave-wl2h krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:48.390429 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:38311
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:48.473672 22248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:48.473937 22248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:48.473975 22248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:48.475129 22248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:48.475201 22248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:48.475227 22248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:48.475256 22248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:48.475286 22248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:48.476771 22248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:48.477067 22248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:48.477298 22248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:48.482354 22253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:48.482333 22254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:48.482334 22256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:48.482369 22248 server_base.cc:1034] running on GCE node
I20260809 16:43:48.482780 22248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:48.483117 22248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:48.484311 22248 hybrid_clock.cc:648] HybridClock initialized: now 1786293828484296 us; error 28 us; skew 500 ppm
Aug 09 16:43:48 dist-test-slave-wl2h krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293828, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:48.486904 22248 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:48.487411 22248 webserver.cc:460] Webserver started at http://127.28.62.190:39251/ using document root <none> and password file <none>
I20260809 16:43:48.487656 22248 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:48.487726 22248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:48.487869 22248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:48.488932 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "95b04503d17a415293208c2e21c6afc9"
format_stamp: "Formatted at 2026-08-09 16:43:48 on dist-test-slave-wl2h"
server_key: "98ee745eed3eb4ddb1062c909344f93e"
server_key_iv: "30a33da3d8b1a6a7707544a1cb2a4eca"
server_key_version: "encryptionkey@0"
I20260809 16:43:48.489302 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "95b04503d17a415293208c2e21c6afc9"
format_stamp: "Formatted at 2026-08-09 16:43:48 on dist-test-slave-wl2h"
server_key: "98ee745eed3eb4ddb1062c909344f93e"
server_key_iv: "30a33da3d8b1a6a7707544a1cb2a4eca"
server_key_version: "encryptionkey@0"
I20260809 16:43:48.490653 22248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:48.491515 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:48.491715 22248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 16:43:48.491837 22248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "95b04503d17a415293208c2e21c6afc9"
format_stamp: "Formatted at 2026-08-09 16:43:48 on dist-test-slave-wl2h"
server_key: "98ee745eed3eb4ddb1062c909344f93e"
server_key_iv: "30a33da3d8b1a6a7707544a1cb2a4eca"
server_key_version: "encryptionkey@0"
I20260809 16:43:48.491923 22248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:48.504326 22248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:48.506629 22248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:48.506803 22248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:48.511348 22248 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:38311
I20260809 16:43:48.511392 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:38311 every 8 connection(s)
I20260809 16:43:48.511777 22248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:48.512674 22324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:48.514976 22324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9: Bootstrap starting.
I20260809 16:43:48.515597 22324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:48.515888 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22248
I20260809 16:43:48.515898 22324 log.cc:824] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:48.516006 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:48.516207 28922 external_mini_cluster.cc:1428] Setting key b2c45e74c7149ef79b2c06bab96ed314
I20260809 16:43:48.516620 22324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9: No bootstrap required, opened a new log
I20260809 16:43:48.518934 22324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } }
I20260809 16:43:48.519150 22324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:48.519243 22324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b04503d17a415293208c2e21c6afc9, State: Initialized, Role: FOLLOWER
I20260809 16:43:48.519390 22324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } }
I20260809 16:43:48.519481 22324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:48.519513 22324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:48.519562 22324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:48.520293 22324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } }
I20260809 16:43:48.520412 22324 leader_election.cc:302] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b04503d17a415293208c2e21c6afc9; no voters:
I20260809 16:43:48.520601 22324 leader_election.cc:288] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:48.520627 22330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:48.520746 22330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 95b04503d17a415293208c2e21c6afc9, State: Running, Role: LEADER
I20260809 16:43:48.520879 22324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:48.520884 22330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } }
I20260809 16:43:48.521304 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b04503d17a415293208c2e21c6afc9. Latest consensus state: current_term: 1 leader_uuid: "95b04503d17a415293208c2e21c6afc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } } }
I20260809 16:43:48.521430 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b04503d17a415293208c2e21c6afc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b04503d17a415293208c2e21c6afc9" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 38311 } } }
I20260809 16:43:48.521827 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:48.522177 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:48.522305 22342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:48.522787 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:48.516757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44453 (local address 127.28.62.190:38311)
0809 16:43:48.516962 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:48.516972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:48.517007 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:48.517825 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:48.517833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:48.517880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:48.518027 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:48.518286 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:48.519272 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:48.520428 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:48.521010 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:48.521128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:48.521930 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:48.521934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:48.521934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:48.522160 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:48.522200 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:48.522215 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:48.522285 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:48.522500 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 16:43:48.522574 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260809 16:43:48.522857 22342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:48.523988 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:48.524631 22342 catalog_manager.cc:1347] Generated new cluster ID: 26275f3b054e44cfa1134b40ee08df88
I20260809 16:43:48.524688 22342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:48.528905 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:48.525577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48485 (local address 127.28.62.190:38311)
0809 16:43:48.525620 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:48.525621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:48.525725 (+ 104us) server_negotiation.cc:407] Connection header received
0809 16:43:48.525754 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:48.525755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:48.525766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:48.525834 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:48.525938 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:48.526670 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:48.527632 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:48.527930 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:48.528005 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:48.528673 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:48.528674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:48.528674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:48.528679 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:48.528694 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:48.528699 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:48.528762 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:48.528809 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 16:43:48.528833 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:48.532418 22342 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:48.533048 22342 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:48.539692 22342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b04503d17a415293208c2e21c6afc9: Generated new TSK 0
I20260809 16:43:48.539907 22342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:48.543605 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22248
2026-08-09T16:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:48 dist-test-slave-wl2h krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:48 dist-test-slave-wl2h krb5kdc[22361](info): set up 2 sockets
Aug 09 16:43:48 dist-test-slave-wl2h krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260809 16:43:49.695670 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:49 dist-test-slave-wl2h krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:49.830278 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40589
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:33223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:49.913821 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:49.914093 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:49.914134 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:49.915403 22376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:49.915475 22376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:49.915503 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:49.915524 22376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:49.915544 22376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:49.917104 22376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:33223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:40589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:49.917407 22376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:49.917654 22376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:49.922910 22385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:49.922926 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:49.922923 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:49.923307 22376 server_base.cc:1034] running on GCE node
I20260809 16:43:49.923592 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:49.923952 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:49.925143 22376 hybrid_clock.cc:648] HybridClock initialized: now 1786293829925121 us; error 40 us; skew 500 ppm
Aug 09 16:43:49 dist-test-slave-wl2h krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293829, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:49.927937 22376 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:49.928373 22376 webserver.cc:460] Webserver started at http://127.28.62.190:46289/ using document root <none> and password file <none>
I20260809 16:43:49.928592 22376 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:49.928633 22376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:49.928735 22376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:49.929800 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "5b413f1e2ecb4d77bae266e40c414014"
format_stamp: "Formatted at 2026-08-09 16:43:49 on dist-test-slave-wl2h"
server_key: "ec4cdc2503b15650f7ab4898dfcc9015"
server_key_iv: "5977dc1ffb7fc7195fd379e640342e53"
server_key_version: "encryptionkey@0"
I20260809 16:43:49.930099 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "5b413f1e2ecb4d77bae266e40c414014"
format_stamp: "Formatted at 2026-08-09 16:43:49 on dist-test-slave-wl2h"
server_key: "ec4cdc2503b15650f7ab4898dfcc9015"
server_key_iv: "5977dc1ffb7fc7195fd379e640342e53"
server_key_version: "encryptionkey@0"
I20260809 16:43:49.931635 22376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260809 16:43:49.932427 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:49.932684 22376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:49.932796 22376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "5b413f1e2ecb4d77bae266e40c414014"
format_stamp: "Formatted at 2026-08-09 16:43:49 on dist-test-slave-wl2h"
server_key: "ec4cdc2503b15650f7ab4898dfcc9015"
server_key_iv: "5977dc1ffb7fc7195fd379e640342e53"
server_key_version: "encryptionkey@0"
I20260809 16:43:49.932870 22376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:49.936354 22376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:49.938441 22376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:49.938699 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:49.943313 22376 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:40589
I20260809 16:43:49.943377 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:40589 every 8 connection(s)
I20260809 16:43:49.943725 22376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:49.944770 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:49.946218 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22376
I20260809 16:43:49.946344 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:49.946552 28922 external_mini_cluster.cc:1428] Setting key c666f60f299b7c7add8162b2f5e6ba3f
I20260809 16:43:49.948040 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014: Bootstrap starting.
I20260809 16:43:49.948786 22454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:49.949095 22454 log.cc:824] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:49.949335 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:49.947162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56347 (local address 127.28.62.190:40589)
0809 16:43:49.947365 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:49.947374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:49.947407 (+ 33us) server_negotiation.cc:407] Connection header received
0809 16:43:49.947867 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:49.947873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:49.947900 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:49.948052 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:49.948809 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:49.948813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:49.948813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:49.948818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:49.948863 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:49.948889 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:49.948969 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:49.949098 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 16:43:49.949144 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260809 16:43:49.949870 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014: No bootstrap required, opened a new log
I20260809 16:43:49.950392 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:49.952209 22454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } }
I20260809 16:43:49.952414 22454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 16:43:49.952462 22454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b413f1e2ecb4d77bae266e40c414014, State: Initialized, Role: FOLLOWER
I20260809 16:43:49.952589 22454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } }
I20260809 16:43:49.952739 22454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 16:43:49.952818 22454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 16:43:49.952874 22454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:49.953071 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:49.951913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35735 (local address 127.28.62.190:40589)
0809 16:43:49.951975 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:49.951977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:49.952119 (+ 142us) server_negotiation.cc:407] Connection header received
0809 16:43:49.952167 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:49.952168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:49.952173 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:49.952235 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:49.952830 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:49.952832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:49.952832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 16:43:49.952837 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:49.952853 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 16:43:49.952858 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:49.952928 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:49.952986 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 16:43:49.953001 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 16:43:49.953701 22454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } }
I20260809 16:43:49.953825 22454 leader_election.cc:302] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b413f1e2ecb4d77bae266e40c414014; no voters:
I20260809 16:43:49.954015 22454 leader_election.cc:288] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 16:43:49.954056 22466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:49.954222 22466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [term 1 LEADER]: Becoming Leader. State: Replica: 5b413f1e2ecb4d77bae266e40c414014, State: Running, Role: LEADER
I20260809 16:43:49.954305 22454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:49.954339 22466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } }
I20260809 16:43:49.954717 22466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b413f1e2ecb4d77bae266e40c414014. Latest consensus state: current_term: 1 leader_uuid: "5b413f1e2ecb4d77bae266e40c414014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } } }
I20260809 16:43:49.954728 22467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b413f1e2ecb4d77bae266e40c414014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b413f1e2ecb4d77bae266e40c414014" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 40589 } } }
I20260809 16:43:49.954876 22466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:49.954890 22467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:49.955163 22475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:49.955721 22475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:49.957244 22475 catalog_manager.cc:1347] Generated new cluster ID: c2553a09a4e1492aa52c266863b114fa
I20260809 16:43:49.957312 22475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:49.976468 22475 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:49.976663 22475 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:49.981585 22475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b413f1e2ecb4d77bae266e40c414014: Generated new TSK 0
I20260809 16:43:49.981796 22475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:49.986176 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22376
2026-08-09T16:43:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1443 ms)
[----------] 18 tests from AuthTokenIssuingTest (25970 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:50 dist-test-slave-wl2h krb5kdc[22488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:50 dist-test-slave-wl2h krb5kdc[22488](info): set up 2 sockets
Aug 09 16:43:50 dist-test-slave-wl2h krb5kdc[22488](info): commencing operation
krb5kdc: starting...
W20260809 16:43:51.116529 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:51.250137 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44761
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:51.333400 22504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:51.333688 22504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:51.333722 22504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:51.333770 22504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:51.334954 22504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:51.335064 22504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:51.335104 22504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:51.335140 22504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:51.335165 22504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:51.336737 22504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:44761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:51.337034 22504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:51.337287 22504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:51.342367 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.342381 22512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.342367 22514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:51.342650 22504 server_base.cc:1034] running on GCE node
I20260809 16:43:51.342906 22504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:51.343232 22504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:51.344483 22504 hybrid_clock.cc:648] HybridClock initialized: now 1786293831344473 us; error 37 us; skew 500 ppm
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:51.350549 22504 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:51.350988 22504 webserver.cc:460] Webserver started at http://127.28.62.190:40345/ using document root <none> and password file <none>
I20260809 16:43:51.351269 22504 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:51.351316 22504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:51.351418 22504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:51.352484 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "d0f86946850847fa974ef944fb2b1690"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "58367514cfb8ddd597cd64bfd8dcbdaf"
server_key_iv: "53d2d01691fc77fe41636c8e9475f188"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.352790 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "d0f86946850847fa974ef944fb2b1690"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "58367514cfb8ddd597cd64bfd8dcbdaf"
server_key_iv: "53d2d01691fc77fe41636c8e9475f188"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.354104 22504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:51.355099 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:51.355324 22504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:51.355434 22504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "d0f86946850847fa974ef944fb2b1690"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "58367514cfb8ddd597cd64bfd8dcbdaf"
server_key_iv: "53d2d01691fc77fe41636c8e9475f188"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.355527 22504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:51.365522 22504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:51.368255 22504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:51.368381 22504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:51.372720 22504 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:44761
I20260809 16:43:51.372777 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:44761 every 8 connection(s)
I20260809 16:43:51.373114 22504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:51.374235 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:51.375934 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22504
I20260809 16:43:51.376081 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:51.376315 28922 external_mini_cluster.cc:1428] Setting key 721c5f3ee592f7ffbde74e95f2f69785
I20260809 16:43:51.376233 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.62.189@KRBTEST.COM; defaulting to no policy
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.189@KRBTEST.COM" created.
I20260809 16:43:51.390725 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.377300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44761 (local address 127.0.0.1:44554)
0809 16:43:51.377543 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.377621 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.378442 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.378957 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.378961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.379439 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.379713 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.379721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.382689 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.382690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.383948 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.383951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.384743 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.384745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.384835 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.385403 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.385414 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.387244 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.388886 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.388894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.388896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.389130 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.389485 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.389486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.389487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.389605 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.390125 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.390131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.390194 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.390288 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.390504 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":73,"thread_start_us":115,"threads_started":1}
I20260809 16:43:51.390727 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.377545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.28.62.190:44761)
0809 16:43:51.378611 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.378617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.378649 (+ 32us) server_negotiation.cc:407] Connection header received
0809 16:43:51.378738 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.378741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.378765 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.378878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.381128 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.382561 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.384051 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.384601 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.384723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.387341 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.387349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.387349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.387356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.388810 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.389232 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.389234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.389234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.389315 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.389695 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.389696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.389697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.390017 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.390108 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.390335 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.390457 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":77,"threads_started":1}
I20260809 16:43:51.391527 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.391923 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.391801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41664)
0809 16:43:51.391849 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.391899 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 16:43:51.392112 22524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
I20260809 16:43:51.392616 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.392525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41676)
0809 16:43:51.392571 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.392598 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 16:43:51.392730 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:37635: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
I20260809 16:43:51.410485 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:37635
--webserver_interface=127.28.62.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:43:51.441990 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } attempt: 1
I20260809 16:43:51.442607 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.442457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41688)
0809 16:43:51.442528 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.442576 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 16:43:51.442988 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.442898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41696)
0809 16:43:51.442941 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.442969 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 16:43:51.443149 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:37635: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
W20260809 16:43:51.497874 22593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:51.498117 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:51.498136 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:51.498159 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:51.499305 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:51.499347 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:51.499361 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:51.499373 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:51.499385 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:51.500864 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.62.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:51.501084 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:51.501287 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:51.506538 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.506546 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.506546 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:51.506831 22593 server_base.cc:1034] running on GCE node
I20260809 16:43:51.507143 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:51.507469 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:51.508641 22593 hybrid_clock.cc:648] HybridClock initialized: now 1786293831508624 us; error 32 us; skew 500 ppm
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:51.511323 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } attempt: 2
I20260809 16:43:51.511559 22593 init.cc:377] Logged in from keytab as kudu/127.28.62.189@KRBTEST.COM (short username kudu)
I20260809 16:43:51.511732 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.511599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41710)
0809 16:43:51.511658 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.511691 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 16:43:51.512082 22593 webserver.cc:460] Webserver started at http://127.28.62.189:40031/ using document root <none> and password file <none>
I20260809 16:43:51.512095 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.511985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41726)
0809 16:43:51.512029 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.512068 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 16:43:51.512176 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:37635: Network error: Client connection negotiation failed: client connection to 127.28.62.189:37635: connect: Connection refused (error 111)
I20260809 16:43:51.512375 22593 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:51.512426 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:51.512571 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:51.513722 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data/instance:
uuid: "3017a2bfcd124f88a9369bb335ec2ddd"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "eb3c887dfd152881c5067ad0722214f4"
server_key_iv: "6ad2dc54050118598ca055b2588ff3a9"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.514168 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal/instance:
uuid: "3017a2bfcd124f88a9369bb335ec2ddd"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "eb3c887dfd152881c5067ad0722214f4"
server_key_iv: "6ad2dc54050118598ca055b2588ff3a9"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.515715 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:51.516551 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:51.516724 22593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260809 16:43:51.516830 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "3017a2bfcd124f88a9369bb335ec2ddd"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "eb3c887dfd152881c5067ad0722214f4"
server_key_iv: "6ad2dc54050118598ca055b2588ff3a9"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.516906 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:51.533933 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:51.537228 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:51.537410 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:51.543470 22593 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.189:37635
I20260809 16:43:51.543532 22674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.189:37635 every 8 connection(s)
I20260809 16:43:51.543972 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
I20260809 16:43:51.545318 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:51.547004 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22593
I20260809 16:43:51.547171 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-1/wal/instance
I20260809 16:43:51.547334 28922 external_mini_cluster.cc:1428] Setting key c116a257d73f02abef2c50fa58083ede
I20260809 16:43:51.547946 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.62.188@KRBTEST.COM; defaulting to no policy
I20260809 16:43:51.561228 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.549751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.28.62.190:44761)
0809 16:43:51.549801 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.549803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.549811 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:51.549838 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.549839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.549857 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.549916 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.550944 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.551823 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.553496 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.553868 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.553923 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.557265 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.557267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.557268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.557278 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.559129 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.559812 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.559813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.559814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.559887 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.560295 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.560296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.560297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.560503 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.560588 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.560959 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.561088 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 16:43:51.561383 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.548739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44761 (local address 127.0.0.1:44562)
0809 16:43:51.548937 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.548983 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.549592 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.550021 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.550024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.550527 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.550807 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.550813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.551963 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.551965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.553374 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.553379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.554052 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.554053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.554242 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.555038 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.555052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.557141 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.559309 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.559319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.559322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.559686 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.559988 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.559990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.559990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.560190 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.560577 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.560586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.560792 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.560917 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.561117 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.188@KRBTEST.COM" created.
I20260809 16:43:51.562062 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:51.570960 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.562532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41740)
0809 16:43:51.562741 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.562753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.562808 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.562948 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.562949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.563143 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.563197 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.563198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.564058 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.564060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.564869 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.564873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.565644 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.565644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.565725 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.566200 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.566206 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.567647 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.569932 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.569933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.569934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.570112 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.570345 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.570346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.570346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.570387 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.570682 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.570683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.570715 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.570766 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.570859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260809 16:43:51.571399 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.562583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.28.62.189:37635)
0809 16:43:51.562755 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.562760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.562807 (+ 47us) server_negotiation.cc:407] Connection header received
0809 16:43:51.562867 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.562869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.562883 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.562949 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.563283 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.563969 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.564976 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.565491 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.565561 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.567757 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.567771 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.567771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.567788 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.569841 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.570179 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.570183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.570183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.570289 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.570436 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.570437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.570437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.570602 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.570665 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.571229 (+ 564us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.571306 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260809 16:43:51.571942 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.572288 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.572177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60764)
0809 16:43:51.572224 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.572266 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 16:43:51.572394 22614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
I20260809 16:43:51.572767 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.572696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60780)
0809 16:43:51.572727 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.572746 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260809 16:43:51.572846 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:32915: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
I20260809 16:43:51.583709 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:32915
--webserver_interface=127.28.62.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:43:51.603319 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } attempt: 3
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:51.610723 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.603806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.28.62.189:37635)
0809 16:43:51.603879 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.603881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.603912 (+ 31us) server_negotiation.cc:407] Connection header received
0809 16:43:51.603947 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.603948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.603959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.604013 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.604384 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.604898 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.605606 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.605831 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.605908 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.607967 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.607969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.607969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.607982 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.609556 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.609918 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.609920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.609920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.609979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.610183 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.610184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.610184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.610310 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.610379 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.610538 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.610620 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 16:43:51.610730 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.603730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41756)
0809 16:43:51.603790 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.603807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.603875 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.604031 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.604032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.604227 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.604298 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.604300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.604981 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.604982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.605517 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.605519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.605905 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.605906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.605988 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.606559 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.606565 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.607866 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.609651 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.609653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.609653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.609845 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.610045 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.610047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.610047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.610113 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.610383 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.610384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.610432 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.610513 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.610620 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260809 16:43:51.611451 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.611869 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.611718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60788)
0809 16:43:51.611804 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.611841 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 16:43:51.612439 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.612334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60798)
0809 16:43:51.612381 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.612409 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 16:43:51.612528 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:32915: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
I20260809 16:43:51.622076 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } attempt: 1
I20260809 16:43:51.622589 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.622448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60804)
0809 16:43:51.622513 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.622557 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 16:43:51.622964 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.622866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60820)
0809 16:43:51.622911 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.622943 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 16:43:51.623256 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:32915: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
I20260809 16:43:51.643682 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } attempt: 1
I20260809 16:43:51.644603 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.644057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60836)
0809 16:43:51.644523 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.644569 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
I20260809 16:43:51.645032 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.644910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60848)
0809 16:43:51.644976 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.645008 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 16:43:51.645270 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:32915: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
W20260809 16:43:51.673164 22683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:51.673444 22683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:51.673475 22683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:51.673519 22683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:51.674693 22683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:51.674769 22683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:51.674795 22683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:51.674823 22683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:51.674853 22683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:51.676429 22683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:44761,127.28.62.189:37635,127.28.62.188:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.62.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:51.676725 22683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:51.676970 22683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:51.681993 22690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.681975 22692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:51.681962 22689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:51.682341 22683 server_base.cc:1034] running on GCE node
I20260809 16:43:51.682546 22683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:51.682803 22683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:51.683951 22683 hybrid_clock.cc:648] HybridClock initialized: now 1786293831683935 us; error 29 us; skew 500 ppm
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:51.686525 22683 init.cc:377] Logged in from keytab as kudu/127.28.62.188@KRBTEST.COM (short username kudu)
I20260809 16:43:51.686939 22683 webserver.cc:460] Webserver started at http://127.28.62.188:39867/ using document root <none> and password file <none>
I20260809 16:43:51.687191 22683 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:51.687239 22683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:51.687333 22683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:51.688402 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data/instance:
uuid: "0f4daa4c068b4bd29777c301f304ae98"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "a9eec1e701f547d55ca31abd44d3f2e3"
server_key_iv: "33abdbd2c579963cb1a734f26e884903"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.688726 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal/instance:
uuid: "0f4daa4c068b4bd29777c301f304ae98"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "a9eec1e701f547d55ca31abd44d3f2e3"
server_key_iv: "33abdbd2c579963cb1a734f26e884903"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.690021 22683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 16:43:51.690749 22699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:51.690937 22683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:51.691056 22683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "0f4daa4c068b4bd29777c301f304ae98"
format_stamp: "Formatted at 2026-08-09 16:43:51 on dist-test-slave-wl2h"
server_key: "a9eec1e701f547d55ca31abd44d3f2e3"
server_key_iv: "33abdbd2c579963cb1a734f26e884903"
server_key_version: "encryptionkey@0"
I20260809 16:43:51.691147 22683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:51.691406 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } attempt: 2
I20260809 16:43:51.691833 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.691682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60860)
0809 16:43:51.691764 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.691803 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 16:43:51.692229 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.692126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60870)
0809 16:43:51.692174 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.692201 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 16:43:51.692332 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:32915: Network error: Client connection negotiation failed: client connection to 127.28.62.188:32915: connect: Connection refused (error 111)
I20260809 16:43:51.709450 22683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:51.711612 22683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:51.711769 22683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:51.715924 22683 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.188:32915
I20260809 16:43:51.715996 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.188:32915 every 8 connection(s)
I20260809 16:43:51.716341 22683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
I20260809 16:43:51.717355 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:51.718895 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.720410 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } attempt: 2
I20260809 16:43:51.721091 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 22683
I20260809 16:43:51.721210 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786293555794774-28922-0/minicluster-data/master-2/wal/instance
I20260809 16:43:51.721421 28922 external_mini_cluster.cc:1428] Setting key 83c4ebcd2bdf6dff768930976ef9d8c9
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:51.730321 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.720724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60882)
0809 16:43:51.720779 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.720791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.720827 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.721183 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.721184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.721382 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.721425 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.721427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.723003 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.723004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.723773 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.723775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.724482 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.724483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.724544 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.724922 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.724928 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.726493 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.728885 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.728886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.728887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.729057 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.729364 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.729365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.729418 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.729866 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.729867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.729924 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.730014 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.730123 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 16:43:51.730542 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.719717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.28.62.190:44761)
0809 16:43:51.719782 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.719785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.720005 (+ 220us) server_negotiation.cc:407] Connection header received
0809 16:43:51.720592 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.720594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.720607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.720663 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.721598 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.722319 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.723338 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.723623 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.723663 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.726725 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.726726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.726726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.726733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.728684 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729244 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.729244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.729245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.729298 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729572 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.729573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.729573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.729758 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.729832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.730118 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.730203 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 16:43:51.730762 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.719674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:44761 (local address 127.0.0.1:44570)
0809 16:43:51.719883 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.719919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.720415 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.720734 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.720752 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.721205 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.721477 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.721483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.722395 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.722396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.723238 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.723240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.723764 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.723765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.723909 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.724475 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.724485 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.726645 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.728764 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.728772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.728775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.729118 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.729369 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.729370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.729491 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.729861 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.729866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.729929 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.730295 (+ 366us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.730395 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":134,"threads_started":1}
I20260809 16:43:51.730827 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.720808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.28.62.188:32915)
0809 16:43:51.721008 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.721013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.721039 (+ 26us) server_negotiation.cc:407] Connection header received
0809 16:43:51.721060 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.721061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.721081 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.721188 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.721523 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.722903 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.723850 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.724348 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.724456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.726587 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.726598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.726599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.726607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.728829 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729155 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.729158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.729159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.729275 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.729482 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.729483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.729483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.729766 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.729876 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.730075 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.730212 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":168,"thread_start_us":139,"threads_started":1}
I20260809 16:43:51.731288 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.731427 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.721935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.28.62.188:32915)
0809 16:43:51.722055 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.722057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.722064 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:51.722082 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.722083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.722091 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.722172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.723068 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.723561 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.724918 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.725128 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.725215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.727843 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.727844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.727844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.727852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.729716 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.730123 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.730124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.730124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.730184 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.730403 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.730404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.730404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.730546 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.730688 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.731187 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.731271 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260809 16:43:51.733249 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: Bootstrap starting.
I20260809 16:43:51.734014 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:51.734349 22586 log.cc:824] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:51.735298 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: No bootstrap required, opened a new log
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:51.737491 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.737672 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f86946850847fa974ef944fb2b1690, State: Initialized, Role: FOLLOWER
I20260809 16:43:51.737773 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.738128 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:51.739246 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
W20260809 16:43:51.740104 22782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:51.740165 22782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:51.740556 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.731995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.28.62.190:44761)
0809 16:43:51.732034 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.732036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.732054 (+ 18us) server_negotiation.cc:407] Connection header received
0809 16:43:51.732074 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.732075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.732084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.732149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.732608 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.733303 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.734296 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.734545 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.734624 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.736951 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.736952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.736953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.736960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.738712 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.739159 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.739160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.739160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.739232 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.739532 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.739532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.739533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.739717 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.740348 (+ 631us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.740393 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.740468 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:51.739377 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.742882 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.731759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.28.62.189:37635)
0809 16:43:51.734670 (+ 2911us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.734673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.734681 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:51.734708 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.734709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.734721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.734784 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.735220 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.735887 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.736843 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.737110 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.737189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.739676 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.739678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.739678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.739687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.741413 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.741961 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.741962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.741963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.742027 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.742282 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.742283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.742284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.742443 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.742483 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.742712 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.742790 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2877}
I20260809 16:43:51.743183 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:37635 (local address 127.0.0.1:41758)
0809 16:43:51.731619 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.731629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.731665 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.734846 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.734847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.735006 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.735103 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.735105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.735992 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.735993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.736748 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.736750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.737421 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.737423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.737491 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.738043 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.738049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.739573 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.741656 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.741657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.741657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.741871 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.742131 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.742132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.742133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.742203 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.742530 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.742530 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.742584 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.742964 (+ 380us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.743093 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260809 16:43:51.743588 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:51.741099 22504 master_runner.cc:186] Error getting master registration for 127.28.62.189:37635: OK, instance_id { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" instance_seqno: 1786293831542147 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:43:51.744344 22504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:51.750512 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.743840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60896)
0809 16:43:51.743888 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.743900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.743942 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.744104 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.744104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.744253 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.744286 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.744287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.745015 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.745016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.745481 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.745482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.745976 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.745977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.746027 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.746414 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.746418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.747685 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.749360 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.749361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.749362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.749575 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.749776 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.749777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.749777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.749825 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.750168 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.750168 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.750209 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.750278 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.750377 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 16:43:51.750756 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.743906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.28.62.188:32915)
0809 16:43:51.743937 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.743938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.743946 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:51.744009 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.744010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.744019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.744044 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.744470 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.744939 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.745561 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.745887 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.745975 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.747780 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.747781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.747782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.747788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.749282 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.749660 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.749661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.749661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.749712 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.749889 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.749890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.749890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.750068 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.750123 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.750323 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.750441 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":16}
I20260809 16:43:51.751515 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.741497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.28.62.189:37635)
0809 16:43:51.741670 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.741672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.741680 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:51.741708 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.741709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.741719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.742381 (+ 662us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.742441 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.744003 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.744977 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.745302 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.745600 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.748391 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.748392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.748392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.748399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.750003 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.750409 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.750411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.750411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.750469 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.750809 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.750810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.750810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.751066 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.751222 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.751307 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.751409 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260809 16:43:51.752418 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: Bootstrap starting.
I20260809 16:43:51.752933 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:51.753173 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:51.753453 22762 log.cc:824] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:51.753872 22555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d0f86946850847fa974ef944fb2b1690"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584
I20260809 16:43:51.754032 22555 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 16:43:51.754091 22555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:51.754290 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: No bootstrap required, opened a new log
I20260809 16:43:51.755379 22555 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.755638 22555 leader_election.cc:288] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [CANDIDATE]: Term 1 election: Requested vote from peers 3017a2bfcd124f88a9369bb335ec2ddd (127.28.62.189:37635), 0f4daa4c068b4bd29777c301f304ae98 (127.28.62.188:32915)
I20260809 16:43:51.757006 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.757179 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4daa4c068b4bd29777c301f304ae98, State: Initialized, Role: FOLLOWER
I20260809 16:43:51.757277 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.757622 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:51.757676 22732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0f86946850847fa974ef944fb2b1690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f4daa4c068b4bd29777c301f304ae98"
I20260809 16:43:51.757781 22732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:51.757573 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.758224 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.759227 22732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f86946850847fa974ef944fb2b1690 in term 1.
I20260809 16:43:51.759465 22524 leader_election.cc:302] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4daa4c068b4bd29777c301f304ae98, d0f86946850847fa974ef944fb2b1690; no voters:
I20260809 16:43:51.760526 22770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:51.760726 22770 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [term 1 LEADER]: Becoming Leader. State: Replica: d0f86946850847fa974ef944fb2b1690, State: Running, Role: LEADER
I20260809 16:43:51.760829 22770 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.761301 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f86946850847fa974ef944fb2b1690. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.761389 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: This master's current role is: LEADER
W20260809 16:43:51.762147 22811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:51.762203 22811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:51.762995 22683 master_runner.cc:186] Error getting master registration for 127.28.62.189:37635: OK, instance_id { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" instance_seqno: 1786293831542147 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:43:51.765350 22683 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 16:43:51.767457 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.756928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.28.62.189:37635)
0809 16:43:51.758156 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.758158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.759257 (+ 1099us) server_negotiation.cc:407] Connection header received
0809 16:43:51.759328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.759329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.759339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.759383 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.760063 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.760807 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.761896 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.762226 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.762285 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.763920 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.763922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.763922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.763930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.765854 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.766271 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.766273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.766273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.766330 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.766512 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.766512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.766513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.766639 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.766743 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.767100 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.767270 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202}
I20260809 16:43:51.768538 22815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:51.768817 22815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:51.770684 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.756974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.28.62.190:44761)
0809 16:43:51.757014 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.757017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.757234 (+ 217us) server_negotiation.cc:407] Connection header received
0809 16:43:51.757594 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.757595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.757607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.757639 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.759700 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.760473 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.764171 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.764467 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.764513 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.766771 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.766772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.766773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.766782 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.769201 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.769651 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.769652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.769653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.769731 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.769941 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.769942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.769942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.770072 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.770161 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.770494 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.770594 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:51.771606 22732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [term 1 FOLLOWER]: Refusing update from remote peer d0f86946850847fa974ef944fb2b1690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:51.771967 22812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:51.772936 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.757189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.28.62.188:32915)
0809 16:43:51.757229 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.757231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.762459 (+ 5228us) server_negotiation.cc:407] Connection header received
0809 16:43:51.762554 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.762555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.762565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.762607 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.766048 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.766896 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.767997 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.768294 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.768376 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.769812 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.769813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.769814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.769821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.771517 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.771953 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.771955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.771955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.772019 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.772298 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.772299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.772300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.772436 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.772530 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.772763 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.772849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:51.773459 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f86946850847fa974ef944fb2b1690. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.773550 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.774654 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.774746 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:51.775115 22815 catalog_manager.cc:1347] Generated new cluster ID: 529bb1f06d01418288e4ce3d6a984f23
I20260809 16:43:51.775127 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.775192 22815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:51.775200 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.775470 22817 mvcc.cc:205] Tried to move back new op lower bound from 7316659534930702336 to 7316659534893297664. Current Snapshot: MvccSnapshot[applied={T|T < 7316659534930702336 or (T in {7316659534930702336})}]
I20260809 16:43:51.775633 22818 mvcc.cc:205] Tried to move back new op lower bound from 7316659534930702336 to 7316659534893297664. Current Snapshot: MvccSnapshot[applied={T|T < 7316659534930702336}]
I20260809 16:43:51.783468 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } attempt: 3
Aug 09 16:43:51 dist-test-slave-wl2h krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:51.792352 22815 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:51.793108 22815 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:51.794240 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.783800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:32915 (local address 127.0.0.1:60906)
0809 16:43:51.783910 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:51.783922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:51.783966 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:51.785178 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:51.785179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:51.785376 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:51.785428 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.785430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.786366 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.786367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.787242 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:51.787246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.787807 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.787808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:51.787867 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.789292 (+ 1425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:51.789297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:51.790867 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:51.792986 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.792987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.792988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.793222 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.793472 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:51.793472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:51.793473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:51.793544 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:51.793879 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:51.793880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:51.793939 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 16:43:51.794001 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 16:43:51.794132 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 16:43:51.794714 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.784038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.28.62.188:32915)
0809 16:43:51.785038 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.785040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.785047 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:51.785066 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.785067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.785077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.785125 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.785512 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.786268 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.787350 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.787684 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.787748 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.790974 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:51.790975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:51.790975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:51.790983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:51.792874 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.793318 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.793319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.793320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.793393 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:51.793630 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:51.793631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:51.793631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:51.793775 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:51.793838 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.794528 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.794626 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980}
I20260809 16:43:51.796303 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: Bootstrap starting.
I20260809 16:43:51.797087 22675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:51.797530 22675 log.cc:824] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:51.804956 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: No bootstrap required, opened a new log
I20260809 16:43:51.807189 22675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.807353 22675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3017a2bfcd124f88a9369bb335ec2ddd, State: Initialized, Role: FOLLOWER
I20260809 16:43:51.807443 22675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } }
I20260809 16:43:51.807809 22675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:51.807844 22648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0f86946850847fa974ef944fb2b1690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3017a2bfcd124f88a9369bb335ec2ddd"
I20260809 16:43:51.807955 22647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:51.808945 22647 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [term 1 FOLLOWER]: Refusing update from remote peer d0f86946850847fa974ef944fb2b1690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:51.809394 22815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690: Generated new TSK 0
I20260809 16:43:51.809621 22815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:51.809736 22648 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f86946850847fa974ef944fb2b1690 in term 1.
I20260809 16:43:51.810417 22812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:51.812674 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.812763 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.812819 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f86946850847fa974ef944fb2b1690. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.812870 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.812906 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:51.812954 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:51.813511 22837 catalog_manager.cc:1259] Loaded cluster ID: 529bb1f06d01418288e4ce3d6a984f23
I20260809 16:43:51.813560 22837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: loading cluster ID for follower catalog manager: success
I20260809 16:43:51.814759 22837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: acquiring CA information for follower catalog manager: success
I20260809 16:43:51.815109 22837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3017a2bfcd124f88a9369bb335ec2ddd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 16:43:51.844544 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.842824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.28.62.189:37635)
0809 16:43:51.842889 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.842891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.843831 (+ 940us) server_negotiation.cc:407] Connection header received
0809 16:43:51.843940 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.843942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.843953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.843985 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.844468 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41800: BlockingRecv error: recv got EOF from 127.0.0.1:41800 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260809 16:43:51.845340 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.843235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.28.62.188:32915)
0809 16:43:51.843279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.843281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.844802 (+ 1521us) server_negotiation.cc:407] Connection header received
0809 16:43:51.844901 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.844902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.844914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.844944 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.845277 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60910: BlockingRecv error: recv got EOF from 127.0.0.1:60910 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260809 16:43:51.846079 22589 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.843562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.28.62.190:44761)
0809 16:43:51.843602 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.843604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.845567 (+ 1963us) server_negotiation.cc:407] Connection header received
0809 16:43:51.845668 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.845669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.845681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.845711 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.846013 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44608: BlockingRecv error: recv got EOF from 127.0.0.1:44608 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:51.851341 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.847724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.28.62.189:37635)
0809 16:43:51.847759 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.847761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.847951 (+ 190us) server_negotiation.cc:407] Connection header received
0809 16:43:51.848057 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.848058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.848117 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.848166 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.848382 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.849175 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.850322 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.850641 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.850683 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.850911 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.851059 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.851123 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.851242 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.851265 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260809 16:43:51.854720 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.853864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.28.62.188:32915)
0809 16:43:51.853910 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.853912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.854176 (+ 264us) server_negotiation.cc:407] Connection header received
0809 16:43:51.854288 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.854289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.854308 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.854338 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.854670 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60912: BlockingRecv error: recv got EOF from 127.0.0.1:60912 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:51.860330 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.856949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.28.62.189:37635)
0809 16:43:51.856984 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.856986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.857233 (+ 247us) server_negotiation.cc:407] Connection header received
0809 16:43:51.857319 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.857319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.857329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.857355 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.857528 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.858276 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.859536 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.859828 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.859870 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.860061 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.860105 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.860133 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.860239 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.860258 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 16:43:51.863632 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.862711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.28.62.188:32915)
0809 16:43:51.862751 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.862754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.863079 (+ 325us) server_negotiation.cc:407] Connection header received
0809 16:43:51.863199 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.863200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.863213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.863242 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.863580 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60928: BlockingRecv error: recv got EOF from 127.0.0.1:60928 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:51.870012 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.866758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.28.62.189:37635)
0809 16:43:51.866806 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.866808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.867071 (+ 263us) server_negotiation.cc:407] Connection header received
0809 16:43:51.867186 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.867187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.867196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.867227 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.867395 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.868138 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.869220 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.869520 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.869560 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.869754 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.869790 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.869817 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.869919 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.869937 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260809 16:43:51.873337 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.872487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.28.62.188:32915)
0809 16:43:51.872527 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.872529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.872778 (+ 249us) server_negotiation.cc:407] Connection header received
0809 16:43:51.872878 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.872879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.872890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.872919 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.873282 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60940: BlockingRecv error: recv got EOF from 127.0.0.1:60940 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260809 16:43:51.882051 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.878747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.28.62.189:37635)
0809 16:43:51.878790 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.878792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.879067 (+ 275us) server_negotiation.cc:407] Connection header received
0809 16:43:51.879183 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.879184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.879195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.879224 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.879397 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.880137 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.881266 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.881561 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.881600 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.881793 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.881826 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.881853 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.881954 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.881974 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260809 16:43:51.885352 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.884483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.28.62.188:32915)
0809 16:43:51.884524 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.884526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.884782 (+ 256us) server_negotiation.cc:407] Connection header received
0809 16:43:51.884880 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.884881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.884893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.884923 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.885299 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60948: BlockingRecv error: recv got EOF from 127.0.0.1:60948 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260809 16:43:51.898010 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.894664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.28.62.189:37635)
0809 16:43:51.894734 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.894736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.895040 (+ 304us) server_negotiation.cc:407] Connection header received
0809 16:43:51.895167 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.895168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.895179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.895208 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.895384 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.896146 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.897229 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.897502 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.897547 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.897750 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.897786 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.897812 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.897907 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.897927 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 16:43:51.901446 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.900597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.28.62.188:32915)
0809 16:43:51.900644 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.900646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.900855 (+ 209us) server_negotiation.cc:407] Connection header received
0809 16:43:51.900956 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.900957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.900969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.901000 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.901388 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60962: BlockingRecv error: recv got EOF from 127.0.0.1:60962 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:51.922713 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.918930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.28.62.189:37635)
0809 16:43:51.918980 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.918982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.919458 (+ 476us) server_negotiation.cc:407] Connection header received
0809 16:43:51.919576 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.919578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.919603 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.919636 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.919850 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.920635 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.921819 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.922113 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.922163 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.922386 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.922424 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.922453 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.922564 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.922628 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260809 16:43:51.926355 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.925407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.28.62.188:32915)
0809 16:43:51.925452 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.925455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.925755 (+ 300us) server_negotiation.cc:407] Connection header received
0809 16:43:51.925858 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.925859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.925872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.925903 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.926294 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60978: BlockingRecv error: recv got EOF from 127.0.0.1:60978 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 16:43:51.963591 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:51.960027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.28.62.189:37635)
0809 16:43:51.960079 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.960081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.960411 (+ 330us) server_negotiation.cc:407] Connection header received
0809 16:43:51.960528 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.960529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.960540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.960570 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:51.960789 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.961523 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.962676 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:51.962980 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:51.963053 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:51.963267 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:51.963303 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:51.963331 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:51.963481 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 16:43:51.963507 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260809 16:43:51.966131 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:51.965480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.28.62.188:32915)
0809 16:43:51.965530 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:51.965531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:51.965598 (+ 67us) server_negotiation.cc:407] Connection header received
0809 16:43:51.965664 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:51.965665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:51.965676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:51.965748 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:51.966074 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60990: BlockingRecv error: recv got EOF from 127.0.0.1:60990 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:52.034250 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.031608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.28.62.189:37635)
0809 16:43:52.031693 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.031695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.031794 (+ 99us) server_negotiation.cc:407] Connection header received
0809 16:43:52.031833 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.031834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.031844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.031913 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.032098 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.032607 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.033443 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.033637 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.033722 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.033977 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.034011 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.034060 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.034164 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.034183 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 16:43:52.036805 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:52.036231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.28.62.188:32915)
0809 16:43:52.036308 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.036310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.036351 (+ 41us) server_negotiation.cc:407] Connection header received
0809 16:43:52.036416 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.036417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.036427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.036475 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:52.036755 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60996: BlockingRecv error: recv got EOF from 127.0.0.1:60996 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 16:43:52.168411 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.166082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.28.62.189:37635)
0809 16:43:52.166189 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.166190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.166204 (+ 14us) server_negotiation.cc:407] Connection header received
0809 16:43:52.166310 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.166311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.166321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.166369 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.166567 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.167120 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.167857 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.168039 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.168092 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.168207 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.168241 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.168280 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.168326 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.168344 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 16:43:52.170852 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:52.170126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.28.62.188:32915)
0809 16:43:52.170205 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.170207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.170268 (+ 61us) server_negotiation.cc:407] Connection header received
0809 16:43:52.170345 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.170346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.170361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.170426 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:52.170779 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60998: BlockingRecv error: recv got EOF from 127.0.0.1:60998 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 16:43:52.286263 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f86946850847fa974ef944fb2b1690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f86946850847fa974ef944fb2b1690" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 44761 } } peers { permanent_uuid: "3017a2bfcd124f88a9369bb335ec2ddd" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 37635 } } peers { permanent_uuid: "0f4daa4c068b4bd29777c301f304ae98" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 32915 } } }
I20260809 16:43:52.286437 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f86946850847fa974ef944fb2b1690 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:52.430559 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.428212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.28.62.189:37635)
0809 16:43:52.428280 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.428294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.428302 (+ 8us) server_negotiation.cc:407] Connection header received
0809 16:43:52.428328 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.428329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.428339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.428415 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.428576 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.429065 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.429854 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.430051 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.430125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.430295 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.430323 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.430381 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.430442 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.430460 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 16:43:52.433257 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:52.432405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.28.62.188:32915)
0809 16:43:52.432481 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.432483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.432518 (+ 35us) server_negotiation.cc:407] Connection header received
0809 16:43:52.432596 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.432597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.432613 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.432676 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:52.433214 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32772: BlockingRecv error: recv got EOF from 127.0.0.1:32772 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 16:43:52.762693 22811 catalog_manager.cc:1259] Loaded cluster ID: 529bb1f06d01418288e4ce3d6a984f23
I20260809 16:43:52.762806 22811 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: loading cluster ID for follower catalog manager: success
I20260809 16:43:52.763706 22811 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: acquiring CA information for follower catalog manager: success
I20260809 16:43:52.763974 22811 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f4daa4c068b4bd29777c301f304ae98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:43:52.949088 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.946536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.28.62.189:37635)
0809 16:43:52.946685 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.946687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.946699 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:52.946805 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.946806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.946817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.946888 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.947052 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.947558 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.948356 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.948558 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.948643 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.948809 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.948841 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.948897 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.948955 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.948982 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260809 16:43:52.953622 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.951032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.28.62.188:32915)
0809 16:43:52.951180 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.951183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.951192 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:52.951253 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.951254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.951317 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.951400 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.951523 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.952012 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.952768 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.953019 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.953093 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.953271 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.953392 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.953479 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.953502 (+ 23us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.953527 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260809 16:43:52.958609 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.955710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.28.62.188:32915)
0809 16:43:52.955864 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.955866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.955972 (+ 106us) server_negotiation.cc:407] Connection header received
0809 16:43:52.956038 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.956039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.956048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.956158 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.956309 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.956790 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.957846 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.958079 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.958154 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.958316 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.958347 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.958416 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.958507 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.958535 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260809 16:43:52.958654 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.955449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.28.62.189:37635)
0809 16:43:52.955504 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.955506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.955659 (+ 153us) server_negotiation.cc:407] Connection header received
0809 16:43:52.955745 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.955746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.955755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.955811 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.955972 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.956727 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.957828 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.958115 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.958154 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.958373 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.958410 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.958461 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.958555 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.958580 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 16:43:52.959206 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:52.955432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.28.62.190:44761)
0809 16:43:52.955642 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:52.955644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:52.955653 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:52.955716 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:52.955717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:52.955776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:52.955870 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:52.956017 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.956770 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.957978 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:52.958316 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:52.958548 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:52.958631 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:52.958766 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:52.958863 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:52.958946 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 16:43:52.958974 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":53,"threads_started":1}
I20260809 16:43:52.961025 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22504
I20260809 16:43:52.969269 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22593
I20260809 16:43:52.977617 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 22683
2026-08-09T16:43:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2990 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 16:43:53 dist-test-slave-wl2h krb5kdc[23007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 16:43:53 dist-test-slave-wl2h krb5kdc[23007](info): set up 2 sockets
Aug 09 16:43:53 dist-test-slave-wl2h krb5kdc[23007](info): commencing operation
krb5kdc: starting...
W20260809 16:43:54.119832 28922 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/test-user.keytab.
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T16:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T16:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.62.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
Entry for principal kudu/127.28.62.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab.
I20260809 16:43:54.250077 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:37075
--webserver_interface=127.28.62.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 16:43:54.332823 23022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:54.333047 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:54.333065 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:54.333086 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:54.334168 23022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:54.334206 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:54.334220 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:54.334233 23022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:54.334245 23022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:54.335794 23022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.190.keytab
--principal=kudu/127.28.62.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.190:37075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.62.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:54.335990 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:54.336175 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:54.341146 23032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.341099 23028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.341176 23029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:54.341465 23022 server_base.cc:1034] running on GCE node
I20260809 16:43:54.341753 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:54.342053 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:54.343214 23022 hybrid_clock.cc:648] HybridClock initialized: now 1786293834343200 us; error 46 us; skew 500 ppm
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:54.345774 23022 init.cc:377] Logged in from keytab as kudu/127.28.62.190@KRBTEST.COM (short username kudu)
I20260809 16:43:54.346248 23022 webserver.cc:460] Webserver started at http://127.28.62.190:45277/ using document root <none> and password file <none>
I20260809 16:43:54.346475 23022 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:54.346539 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:54.346694 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:54.347828 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data/instance:
uuid: "dc9f87da76dd42839f9b1b27b56ced01"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e7984ffb8d90a83ea545f3624f9fcbed"
server_key_iv: "90c2db8c8be280a38edf7556eaf058a5"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.348198 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance:
uuid: "dc9f87da76dd42839f9b1b27b56ced01"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e7984ffb8d90a83ea545f3624f9fcbed"
server_key_iv: "90c2db8c8be280a38edf7556eaf058a5"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.349611 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 16:43:54.350391 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:54.350584 23022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 16:43:54.350699 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal
uuid: "dc9f87da76dd42839f9b1b27b56ced01"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e7984ffb8d90a83ea545f3624f9fcbed"
server_key_iv: "90c2db8c8be280a38edf7556eaf058a5"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.350772 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:54.362660 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:54.365047 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:54.365212 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:54.369426 23022 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.190:37075
I20260809 16:43:54.369463 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.190:37075 every 8 connection(s)
I20260809 16:43:54.369849 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/data/info.pb
I20260809 16:43:54.370913 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:54.372512 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.375643 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 23022
I20260809 16:43:54.375746 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-0/wal/instance
I20260809 16:43:54.375967 28922 external_mini_cluster.cc:1428] Setting key cdb265d1a7ba82148f6fd94865b5e1c7
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:54.383303 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.373154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37075 (local address 127.0.0.1:36496)
0809 16:43:54.373311 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.373362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.373854 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.374107 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.374109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.374383 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.374536 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.374539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.375613 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.375615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.376712 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.376716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.377216 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.377217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.377306 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.377840 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.377865 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.379570 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.381742 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.381749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.381750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.381947 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.382219 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.382219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.382220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.382311 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.382801 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.382806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.382860 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.382932 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.383115 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260809 16:43:54.383334 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.373243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.28.62.190:37075)
0809 16:43:54.373374 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.373378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.373447 (+ 69us) server_negotiation.cc:407] Connection header received
0809 16:43:54.373986 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.373988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.374004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.374080 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.374631 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.375520 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.376807 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.377109 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.377190 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.379655 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.379663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.379664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.379671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.381644 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.382029 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.382033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.382034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.382145 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.382396 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.382397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.382398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.382725 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.382823 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.382994 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.383205 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260809 16:43:54.384138 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.384509 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.384399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46244)
0809 16:43:54.384445 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.384491 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 16:43:54.384676 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
I20260809 16:43:54.385097 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.385012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46250)
0809 16:43:54.385055 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.385080 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 16:43:54.385229 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:45073: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.62.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
Entry for principal kudu/127.28.62.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab.
I20260809 16:43:54.406782 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:45073
--webserver_interface=127.28.62.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:43:54.434499 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } attempt: 1
I20260809 16:43:54.434993 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.434859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46254)
0809 16:43:54.434923 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.434962 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 16:43:54.435381 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.435282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46260)
0809 16:43:54.435328 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.435354 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 16:43:54.435465 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:45073: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
W20260809 16:43:54.490834 23109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:54.491086 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:54.491104 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:54.491127 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:54.492538 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:54.492592 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:54.492611 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:54.492624 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:54.492640 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:54.494318 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.189.keytab
--principal=kudu/127.28.62.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.189:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.62.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:54.494587 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:54.494832 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:54.499909 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.499927 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.499989 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:54.500133 23109 server_base.cc:1034] running on GCE node
I20260809 16:43:54.500478 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:54.500787 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:54.501933 23109 hybrid_clock.cc:648] HybridClock initialized: now 1786293834501919 us; error 34 us; skew 500 ppm
I20260809 16:43:54.503615 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } attempt: 2
I20260809 16:43:54.504063 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.503906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46270)
0809 16:43:54.503981 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.504025 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:54.504494 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.504371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46278)
0809 16:43:54.504422 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.504456 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 16:43:54.504611 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.189:45073: Network error: Client connection negotiation failed: client connection to 127.28.62.189:45073: connect: Connection refused (error 111)
I20260809 16:43:54.504771 23109 init.cc:377] Logged in from keytab as kudu/127.28.62.189@KRBTEST.COM (short username kudu)
I20260809 16:43:54.505221 23109 webserver.cc:460] Webserver started at http://127.28.62.189:46603/ using document root <none> and password file <none>
I20260809 16:43:54.505468 23109 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:54.505517 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:54.505647 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:54.506779 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data/instance:
uuid: "c73f651a4190406c8e0b5596d885e524"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e5fd51f6d65cf543039c4f0e612c7ef5"
server_key_iv: "d001ee7655fcb31e9e30a385cdf6ef31"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.507170 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal/instance:
uuid: "c73f651a4190406c8e0b5596d885e524"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e5fd51f6d65cf543039c4f0e612c7ef5"
server_key_iv: "d001ee7655fcb31e9e30a385cdf6ef31"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.508693 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 16:43:54.509549 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:54.509725 23109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:54.509841 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal
uuid: "c73f651a4190406c8e0b5596d885e524"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "e5fd51f6d65cf543039c4f0e612c7ef5"
server_key_iv: "d001ee7655fcb31e9e30a385cdf6ef31"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.509899 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:54.522807 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:54.524883 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:54.524999 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:54.529143 23109 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.189:45073
I20260809 16:43:54.529196 23186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.189:45073 every 8 connection(s)
I20260809 16:43:54.529541 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/data/info.pb
I20260809 16:43:54.530592 23187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:54.532495 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 23109
I20260809 16:43:54.532619 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-1/wal/instance
I20260809 16:43:54.532457 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.532852 28922 external_mini_cluster.cc:1428] Setting key cfd77bdcfc76df6929b665244b0654df
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:54.542573 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.533416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.28.62.190:37075)
0809 16:43:54.533487 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.533490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.533702 (+ 212us) server_negotiation.cc:407] Connection header received
0809 16:43:54.534165 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.534166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.534185 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.534266 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.534911 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.535483 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.536504 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.536839 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.536906 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.539403 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.539405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.539405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.539414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.541115 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.541544 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.541546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.541546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.541611 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.541865 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.541866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.541867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.542025 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.542099 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.542387 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.542479 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 16:43:54.542686 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.533335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37075 (local address 127.0.0.1:36500)
0809 16:43:54.533562 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.533605 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.534076 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.534336 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.534340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.534588 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.534799 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.534802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.535581 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.535582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.536406 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.536408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.536957 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.536958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.537086 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.537673 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.537682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.539309 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.541208 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.541216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.541217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.541466 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.541683 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.541684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.541684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.541797 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.542104 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.542109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.542232 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.542341 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.542474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.28.62.188@KRBTEST.COM; defaulting to no policy
I20260809 16:43:54.543375 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.62.188@KRBTEST.COM" created.
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:54.551703 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.543643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46286)
0809 16:43:54.543695 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.543708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.543754 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.544076 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.544078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.544205 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.544260 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.544261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.545131 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.545132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.545664 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.545665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.546320 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.546321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.546385 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.546828 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.546833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.548243 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.550552 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.550553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.550553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.550751 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.550983 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.550984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.550984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.551094 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.551443 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.551443 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.551483 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.551541 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.551630 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 16:43:54.551877 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.543716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.28.62.189:45073)
0809 16:43:54.543921 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.543926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.543948 (+ 22us) server_negotiation.cc:407] Connection header received
0809 16:43:54.543969 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.543971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.543984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.544062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.544331 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.545011 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.545740 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.546232 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.546323 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.548329 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.548338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.548338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.548346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.550450 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.550840 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.550843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.550844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.550911 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.551162 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.551164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.551164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.551370 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.551469 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.551580 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.551703 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260809 16:43:54.552573 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.552953 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.552833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60926)
0809 16:43:54.552882 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.552933 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 16:43:54.553092 23130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
I20260809 16:43:54.553584 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.553500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60934)
0809 16:43:54.553533 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.553560 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260809 16:43:54.553701 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:36013: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
Entry for principal kudu/127.28.62.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab.
I20260809 16:43:54.565815 28922 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb9mnon/build/release/bin/kudu
/tmp/dist-test-taskb9mnon/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:36013
--webserver_interface=127.28.62.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.62.148:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 16:43:54.595763 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } attempt: 3
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:54.602689 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.596130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46288)
0809 16:43:54.596197 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.596212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.596270 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.596433 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.596434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.596602 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.596676 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.596678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.597349 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.597351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.597873 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.597875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.598268 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.598269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.598350 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.598818 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.598823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.600083 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.601651 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.601653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.601653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.601828 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.602041 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.602042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.602042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.602102 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.602354 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.602355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.602403 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.602484 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.602584 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 16:43:54.602697 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.596227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.28.62.189:45073)
0809 16:43:54.596302 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.596304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.596316 (+ 12us) server_negotiation.cc:407] Connection header received
0809 16:43:54.596352 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.596353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.596364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.596427 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.596761 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.597259 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.597953 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.598178 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.598267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.600185 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.600186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.600187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.600194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.601569 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.601913 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.601914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.601915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.601971 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.602168 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.602169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.602169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.602284 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.602352 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.602510 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.602588 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:43:54.602943 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } attempt: 1
I20260809 16:43:54.603364 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.603219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60944)
0809 16:43:54.603280 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.603323 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 16:43:54.603372 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.603744 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.603662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60960)
0809 16:43:54.603697 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.603717 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260809 16:43:54.603744 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.603638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60948)
0809 16:43:54.603679 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.603719 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 16:43:54.603822 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:36013: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
I20260809 16:43:54.604116 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.604030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60974)
0809 16:43:54.604076 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.604099 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 16:43:54.604207 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:36013: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
I20260809 16:43:54.635380 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } attempt: 1
I20260809 16:43:54.635910 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.635771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60982)
0809 16:43:54.635840 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.635880 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 16:43:54.636340 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.636254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:60994)
0809 16:43:54.636295 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.636322 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 16:43:54.636453 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:36013: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
W20260809 16:43:54.650537 23195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 16:43:54.650763 23195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 16:43:54.650780 23195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 16:43:54.650800 23195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 16:43:54.651958 23195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 16:43:54.652002 23195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 16:43:54.652016 23195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 16:43:54.652029 23195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 16:43:54.652056 23195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 16:43:54.653519 23195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.62.148:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.62.190:37075,127.28.62.189:45073,127.28.62.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/krb5kdc/kudu_127.28.62.188.keytab
--principal=kudu/127.28.62.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.62.188:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.62.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 09 Aug 2026 14:17:19 UTC on ccycloud.quasar-aikwje.root.comops.site
build id 4576541
I20260809 16:43:54.653753 23195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 16:43:54.653945 23195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 16:43:54.658803 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.658803 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 16:43:54.658830 23204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 16:43:54.659075 23195 server_base.cc:1034] running on GCE node
I20260809 16:43:54.659446 23195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 16:43:54.659708 23195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 16:43:54.660848 23195 hybrid_clock.cc:648] HybridClock initialized: now 1786293834660827 us; error 36 us; skew 500 ppm
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 16:43:54.663298 23195 init.cc:377] Logged in from keytab as kudu/127.28.62.188@KRBTEST.COM (short username kudu)
I20260809 16:43:54.663698 23195 webserver.cc:460] Webserver started at http://127.28.62.188:37789/ using document root <none> and password file <none>
I20260809 16:43:54.663898 23195 fs_manager.cc:359] Metadata directory not provided
I20260809 16:43:54.663938 23195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 16:43:54.664027 23195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 16:43:54.665045 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data/instance:
uuid: "e0dad6191e8945a2a888b5eab7230f16"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "c25a687ca773b587db9390f467a40927"
server_key_iv: "c00df00dc9ffec4c6282d7930ce92c21"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.665344 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal/instance:
uuid: "e0dad6191e8945a2a888b5eab7230f16"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "c25a687ca773b587db9390f467a40927"
server_key_iv: "c00df00dc9ffec4c6282d7930ce92c21"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.666625 23195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260809 16:43:54.667382 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:54.667549 23195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260809 16:43:54.667651 23195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data,/tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal
uuid: "e0dad6191e8945a2a888b5eab7230f16"
format_stamp: "Formatted at 2026-08-09 16:43:54 on dist-test-slave-wl2h"
server_key: "c25a687ca773b587db9390f467a40927"
server_key_iv: "c00df00dc9ffec4c6282d7930ce92c21"
server_key_version: "encryptionkey@0"
I20260809 16:43:54.667735 23195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 16:43:54.671945 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } attempt: 2
I20260809 16:43:54.672386 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.672260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:32778)
0809 16:43:54.672319 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.672357 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 16:43:54.672706 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.672604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:32780)
0809 16:43:54.672648 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.672673 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 16:43:54.672789 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.62.188:36013: Network error: Client connection negotiation failed: client connection to 127.28.62.188:36013: connect: Connection refused (error 111)
I20260809 16:43:54.688551 23195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 16:43:54.690528 23195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 16:43:54.690698 23195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 16:43:54.694717 23195 rpc_server.cc:307] RPC server started. Bound to: 127.28.62.188:36013
I20260809 16:43:54.694754 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.62.188:36013 every 8 connection(s)
I20260809 16:43:54.695195 23195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/data/info.pb
I20260809 16:43:54.696158 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 16:43:54.697731 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.701838 28922 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb9mnon/build/release/bin/kudu as pid 23195
I20260809 16:43:54.701920 28922 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb9mnon/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786293555794774-28922-0/minicluster-data/master-2/wal/instance
I20260809 16:43:54.702126 28922 external_mini_cluster.cc:1428] Setting key e87042568d599fadf1b9bade4d8e230d
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:54.706938 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.698465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.28.62.190:37075)
0809 16:43:54.698522 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.698524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.698759 (+ 235us) server_negotiation.cc:407] Connection header received
0809 16:43:54.699306 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.699307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.699322 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.699392 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.699991 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.700492 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.701428 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.701610 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.701687 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.704063 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.704065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.704065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.704073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.705532 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.705941 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.705943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.705943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.706003 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.706268 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.706269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.706270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.706390 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.706457 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.706755 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.706847 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 16:43:54.707137 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.698424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.190:37075 (local address 127.0.0.1:36508)
0809 16:43:54.698616 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.698664 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.699206 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.699466 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.699469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.699725 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.699918 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.699922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.700567 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.700567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.701349 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.701351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.701704 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.701705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.701805 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.702372 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.702380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.703974 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.705630 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.705637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.705639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.705865 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.706080 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.706080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.706081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.706196 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.706460 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.706468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.706601 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.706719 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.706876 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:54.707718 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
I20260809 16:43:54.710711 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.702547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.28.62.188:36013)
0809 16:43:54.702726 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.702731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.702749 (+ 18us) server_negotiation.cc:407] Connection header received
0809 16:43:54.702776 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.702778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.702801 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.702885 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.703569 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.704284 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.705013 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.705321 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.705399 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.707939 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.707946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.707946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.707953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.709592 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.709923 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.709926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.709926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.709992 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.710181 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.710182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.710182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.710306 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.710443 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.710523 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.710604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260809 16:43:54.711597 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } attempt: 2
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:54.715346 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.707990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.189:45073 (local address 127.0.0.1:46294)
0809 16:43:54.708046 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.708058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.708092 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.708268 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.708269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.708417 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.708474 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.708475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.709226 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.709227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.709924 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.709925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.710415 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.710416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.710490 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.710944 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.710950 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.712451 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.714224 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.714225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.714225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.714434 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.714674 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.714675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.714675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.714742 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.715050 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.715051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.715096 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.715182 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.715260 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 16:43:54.715356 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.708092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.28.62.189:45073)
0809 16:43:54.708143 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.708145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.708152 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:54.708187 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.708188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.708199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.708254 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.708564 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.709144 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.710005 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.710325 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.710405 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.712541 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.712543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.712543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.712550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.714141 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.714529 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.714531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.714531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.714594 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.714811 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.714812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.714812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.714959 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.715058 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.715228 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.715295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.190@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:54.716372 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.190@KRBTEST.COM
I20260809 16:43:54.719766 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.711850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:32800)
0809 16:43:54.711905 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.711914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.711951 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.712106 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.712107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.712269 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.712310 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.712312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.713215 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.713216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.714063 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.714066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.714494 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.714495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.714553 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.714963 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.714967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.716466 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.718568 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.718569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.718570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.718819 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.719070 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.719071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.719071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.719153 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.719437 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.719438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.719483 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.719554 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.719651 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 16:43:54.719776 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.711924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.28.62.188:36013)
0809 16:43:54.711958 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.711960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.711969 (+ 9us) server_negotiation.cc:407] Connection header received
0809 16:43:54.711989 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.711989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.711998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.712382 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.712413 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.713115 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.714167 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.714422 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.714493 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.716705 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.716706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.716706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.716712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.718498 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.718910 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.718911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.718911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.718967 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.719230 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.719231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.719231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.719364 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.719435 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.719595 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.719671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.188@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:54.722131 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: Bootstrap starting.
I20260809 16:43:54.722882 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:54.723166 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.711590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.28.62.190:37075)
0809 16:43:54.712009 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.712011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.712015 (+ 4us) server_negotiation.cc:407] Connection header received
0809 16:43:54.712027 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.712027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.712033 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.712098 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.712592 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.713287 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.714155 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.714437 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.714498 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.717522 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.717523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.717523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.717530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.719212 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.719618 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.719619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.719619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.719684 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.719907 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.719908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.719908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.720043 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.720210 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.720451 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.723057 (+ 2606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260809 16:43:54.723238 23101 log.cc:824] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:54.724257 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: No bootstrap required, opened a new log
I20260809 16:43:54.724435 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.716661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:32804)
0809 16:43:54.716917 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.716925 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.716965 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.717101 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.717102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.717247 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.717274 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.717276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.718142 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.718143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.718795 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.718797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.719694 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.719695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.719742 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.720097 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.720101 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.721458 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.723302 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.723303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.723303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.723517 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.723736 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.723736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.723737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.723806 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.724123 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.724124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.724171 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.724249 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.724342 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260809 16:43:54.724838 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.716715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.28.62.188:36013)
0809 16:43:54.716891 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.716893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.716952 (+ 59us) server_negotiation.cc:407] Connection header received
0809 16:43:54.717018 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.717019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.717028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.717152 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.717344 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.718053 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.718867 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.719591 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.719651 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.721549 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.721551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.721551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.721557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.723166 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.723577 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.723578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.723579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.723655 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.723877 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.723878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.723878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.724047 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.724118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.724623 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.724711 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260809 16:43:54.726052 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.726194 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: Bootstrap starting.
I20260809 16:43:54.726198 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9f87da76dd42839f9b1b27b56ced01, State: Initialized, Role: FOLLOWER
I20260809 16:43:54.726289 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.726569 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.726645 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:54.726869 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:54.726827 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.727224 23283 log.cc:824] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:54.728093 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: No bootstrap required, opened a new log
W20260809 16:43:54.728986 23308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:54.729040 23308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.62.189@KRBTEST.COM
I20260809 16:43:54.730338 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.730490 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dad6191e8945a2a888b5eab7230f16, State: Initialized, Role: FOLLOWER
I20260809 16:43:54.730567 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.730883 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:54.729705 23022 master_runner.cc:186] Error getting master registration for 127.28.62.189:45073: OK, instance_id { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" instance_seqno: 1786293834527996 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:43:54.733132 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.724247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.28.62.189:45073)
0809 16:43:54.724284 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.724285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.724292 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:54.724314 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.724315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.724325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.724371 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.725011 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.725650 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.726603 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.726904 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.726953 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.729882 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.729883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.729884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.729890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.731772 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.732228 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.732230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.732230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.732288 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.732484 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.732484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.732485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.732613 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.732760 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.732944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.733017 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260809 16:43:54.733180 23022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 16:43:54.733526 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.733695 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 16:43:54.734442 23324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:54.734520 23324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:54.734802 28922 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 16:43:54.735771 23069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc9f87da76dd42839f9b1b27b56ced01"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516
I20260809 16:43:54.735877 23069 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 16:43:54.735920 23069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:54.736828 23069 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.737046 23069 leader_election.cc:288] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [CANDIDATE]: Term 1 election: Requested vote from peers c73f651a4190406c8e0b5596d885e524 (127.28.62.189:45073), e0dad6191e8945a2a888b5eab7230f16 (127.28.62.188:36013)
I20260809 16:43:54.737835 23248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc9f87da76dd42839f9b1b27b56ced01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e0dad6191e8945a2a888b5eab7230f16"
I20260809 16:43:54.737947 23248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:54.735802 23195 master_runner.cc:186] Error getting master registration for 127.28.62.189:45073: OK, instance_id { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" instance_seqno: 1786293834527996 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 16:43:54.738078 23195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 16:43:54.738858 23248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9f87da76dd42839f9b1b27b56ced01 in term 1.
I20260809 16:43:54.739254 23044 leader_election.cc:302] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc9f87da76dd42839f9b1b27b56ced01, e0dad6191e8945a2a888b5eab7230f16; no voters:
I20260809 16:43:54.739384 23290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 16:43:54.739522 23290 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [term 1 LEADER]: Becoming Leader. State: Replica: dc9f87da76dd42839f9b1b27b56ced01, State: Running, Role: LEADER
I20260809 16:43:54.739604 23290 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.740008 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9f87da76dd42839f9b1b27b56ced01. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.740095 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:54.740372 23328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 16:43:54.740732 23328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 16:43:54.742460 23248 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [term 1 FOLLOWER]: Refusing update from remote peer dc9f87da76dd42839f9b1b27b56ced01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:54.742825 23290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 16:43:54.744153 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9f87da76dd42839f9b1b27b56ced01. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.744277 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.753065 23335 mvcc.cc:205] Tried to move back new op lower bound from 7316659547102957568 to 7316659547094151168. Current Snapshot: MvccSnapshot[applied={T|T < 7316659547102957568}]
I20260809 16:43:54.753470 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.753568 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: This master's current role is: LEADER
I20260809 16:43:54.754040 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.754145 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.754218 23337 mvcc.cc:205] Tried to move back new op lower bound from 7316659547102957568 to 7316659547094151168. Current Snapshot: MvccSnapshot[applied={T|T < 7316659547102957568}]
I20260809 16:43:54.761586 23328 catalog_manager.cc:1347] Generated new cluster ID: b6d925d6f0e345c8b82e99b327928933
I20260809 16:43:54.761677 23328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 16:43:54.763909 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } attempt: 3
I20260809 16:43:54.768323 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.751784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.28.62.188:36013)
0809 16:43:54.751830 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.751832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.757258 (+ 5426us) server_negotiation.cc:407] Connection header received
0809 16:43:54.757357 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.757358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.757369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.757398 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.758138 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.758898 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.761212 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.761535 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.761574 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.764189 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.764191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.764191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.764207 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.766439 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.767115 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.767116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.767116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.767173 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.767538 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.767539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.767539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.767678 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.767769 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.768125 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.768228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 09 16:43:54 dist-test-slave-wl2h krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786293834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.62.189@KRBTEST.COM for kudu/127.28.62.188@KRBTEST.COM
I20260809 16:43:54.772259 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.764186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.62.188:36013 (local address 127.0.0.1:32828)
0809 16:43:54.764273 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 16:43:54.764284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 16:43:54.764324 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 16:43:54.764577 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 16:43:54.764578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 16:43:54.764759 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 16:43:54.764808 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.764810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.765719 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.765720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.766581 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 16:43:54.766584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.767117 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.767118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 16:43:54.767176 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.767558 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 16:43:54.767563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 16:43:54.769058 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 16:43:54.770942 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.770943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.770944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.771185 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.771519 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 16:43:54.771520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 16:43:54.771520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0809 16:43:54.771594 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 16:43:54.771911 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 16:43:54.771912 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 16:43:54.771969 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 16:43:54.772029 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 16:43:54.772146 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 16:43:54.772584 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.764400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.28.62.188:36013)
0809 16:43:54.764443 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.764445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.764452 (+ 7us) server_negotiation.cc:407] Connection header received
0809 16:43:54.764474 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.764475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.764484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.764533 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.764896 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.765598 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.766669 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.766984 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.767067 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.769307 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.769309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.769309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.769316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.770832 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.771360 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.771362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.771362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.771433 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.771676 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.771677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.771677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.771812 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.771868 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.772424 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.772499 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:54.774184 23187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: Bootstrap starting.
I20260809 16:43:54.774950 23187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: Neither blocks nor log segments found. Creating new log.
I20260809 16:43:54.775295 23187 log.cc:824] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: Log is configured to *not* fsync() on all Append() calls
I20260809 16:43:54.777613 23187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: No bootstrap required, opened a new log
I20260809 16:43:54.778733 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.751399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.28.62.189:45073)
0809 16:43:54.751448 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.751451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.752255 (+ 804us) server_negotiation.cc:407] Connection header received
0809 16:43:54.752424 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.752425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.752437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.752469 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.753957 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.754722 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.756600 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.757390 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.757439 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.775243 (+ 17804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.775245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.775246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.775254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.776886 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.777521 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.777522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.777523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.777581 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.777960 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.777962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.777962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.778111 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.778195 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.778538 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.778636 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:54.779505 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:54.751624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.28.62.190:37075)
0809 16:43:54.751665 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.751667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.754493 (+ 2826us) server_negotiation.cc:407] Connection header received
0809 16:43:54.754592 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.754593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.754604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.754632 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.755417 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.756150 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.758153 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:54.758448 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:54.758489 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:54.775284 (+ 16795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 16:43:54.775286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 16:43:54.775286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 16:43:54.775296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 16:43:54.777090 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.777827 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.777828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.777828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.777908 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 16:43:54.778758 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 16:43:54.778759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 16:43:54.778760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 16:43:54.778917 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 16:43:54.778986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:54.779316 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 16:43:54.779408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:54.779932 23187 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.780098 23187 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73f651a4190406c8e0b5596d885e524, State: Initialized, Role: FOLLOWER
I20260809 16:43:54.780193 23187 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } }
I20260809 16:43:54.780576 23187 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 16:43:54.780596 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc9f87da76dd42839f9b1b27b56ced01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c73f651a4190406c8e0b5596d885e524"
I20260809 16:43:54.780700 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [term 0 FOLLOWER]: Advancing to term 1
I20260809 16:43:54.781713 23154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9f87da76dd42839f9b1b27b56ced01 in term 1.
I20260809 16:43:54.782081 23153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [term 1 FOLLOWER]: Refusing update from remote peer dc9f87da76dd42839f9b1b27b56ced01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 16:43:54.780536 23348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.782292 23348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.783449 23345 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 16:43:54.783828 23362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 16:43:54.783885 23362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 16:43:54.786752 23364 mvcc.cc:205] Tried to move back new op lower bound from 7316659547102957568 to 7316659547094151168. Current Snapshot: MvccSnapshot[applied={T|T < 7316659547102957568}]
I20260809 16:43:54.787063 23348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9f87da76dd42839f9b1b27b56ced01. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.787158 23348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.787194 23348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.787256 23348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 16:43:54.787312 23328 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 16:43:54.788033 23328 catalog_manager.cc:1504] Loading token signing keys...
I20260809 16:43:54.792848 23328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01: Generated new TSK 0
I20260809 16:43:54.793057 23328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 16:43:54.799341 23326 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc9f87da76dd42839f9b1b27b56ced01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9f87da76dd42839f9b1b27b56ced01" member_type: VOTER last_known_addr { host: "127.28.62.190" port: 37075 } } peers { permanent_uuid: "c73f651a4190406c8e0b5596d885e524" member_type: VOTER last_known_addr { host: "127.28.62.189" port: 45073 } } peers { permanent_uuid: "e0dad6191e8945a2a888b5eab7230f16" member_type: VOTER last_known_addr { host: "127.28.62.188" port: 36013 } } }
I20260809 16:43:54.799455 23326 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9f87da76dd42839f9b1b27b56ced01 [sys.catalog]: This master's current role is: LEADER
W20260809 16:43:54.825569 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.823586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.28.62.188:36013)
0809 16:43:54.823637 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.823639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.824754 (+ 1115us) server_negotiation.cc:407] Connection header received
0809 16:43:54.824898 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.824899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.824913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.824948 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.825489 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32836: BlockingRecv error: recv got EOF from 127.0.0.1:32836 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260809 16:43:54.826468 23104 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.824042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.28.62.190:37075)
0809 16:43:54.824088 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.824090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.825870 (+ 1780us) server_negotiation.cc:407] Connection header received
0809 16:43:54.825980 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.825981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.825993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.826028 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.826372 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36532: BlockingRecv error: recv got EOF from 127.0.0.1:36532 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260809 16:43:54.827445 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.824440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.28.62.189:45073)
0809 16:43:54.824492 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.824494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.826744 (+ 2250us) server_negotiation.cc:407] Connection header received
0809 16:43:54.826871 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.826873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.826885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.826919 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.827377 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46330: BlockingRecv error: recv got EOF from 127.0.0.1:46330 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260809 16:43:54.830329 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.829473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.28.62.189:45073)
0809 16:43:54.829515 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.829517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.829737 (+ 220us) server_negotiation.cc:407] Connection header received
0809 16:43:54.829879 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.829880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.829901 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.829935 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.830278 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260809 16:43:54.833652 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.832868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.28.62.189:45073)
0809 16:43:54.832904 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.832906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.833148 (+ 242us) server_negotiation.cc:407] Connection header received
0809 16:43:54.833246 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.833247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.833257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.833287 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.833603 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46348: BlockingRecv error: recv got EOF from 127.0.0.1:46348 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 16:43:54.837741 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.836896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.28.62.189:45073)
0809 16:43:54.836934 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.836936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.837188 (+ 252us) server_negotiation.cc:407] Connection header received
0809 16:43:54.837297 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.837298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.837309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.837340 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.837693 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46362: BlockingRecv error: recv got EOF from 127.0.0.1:46362 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260809 16:43:54.844169 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.843228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.28.62.189:45073)
0809 16:43:54.843271 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.843273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.843550 (+ 277us) server_negotiation.cc:407] Connection header received
0809 16:43:54.843665 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.843666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.843678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.843712 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.844119 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46370: BlockingRecv error: recv got EOF from 127.0.0.1:46370 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260809 16:43:54.854805 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.853671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.28.62.189:45073)
0809 16:43:54.853731 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.853734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.854050 (+ 316us) server_negotiation.cc:407] Connection header received
0809 16:43:54.854175 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.854177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.854190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.854226 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.854714 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46382: BlockingRecv error: recv got EOF from 127.0.0.1:46382 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 16:43:54.873626 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.872495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.28.62.189:45073)
0809 16:43:54.872550 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.872552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.872878 (+ 326us) server_negotiation.cc:407] Connection header received
0809 16:43:54.873010 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.873011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.873025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.873078 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.873560 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46384: BlockingRecv error: recv got EOF from 127.0.0.1:46384 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260809 16:43:54.908149 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.907157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.28.62.189:45073)
0809 16:43:54.907205 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.907208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.907504 (+ 296us) server_negotiation.cc:407] Connection header received
0809 16:43:54.907642 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.907643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.907657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.907688 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.908083 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46400: BlockingRecv error: recv got EOF from 127.0.0.1:46400 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260809 16:43:54.974470 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:54.973671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.28.62.189:45073)
0809 16:43:54.973760 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:54.973763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:54.973881 (+ 118us) server_negotiation.cc:407] Connection header received
0809 16:43:54.973964 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:54.973965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:54.973983 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:54.974051 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:54.974420 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46410: BlockingRecv error: recv got EOF from 127.0.0.1:46410 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 16:43:55.104624 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:55.103821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.28.62.189:45073)
0809 16:43:55.103920 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.103922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.104019 (+ 97us) server_negotiation.cc:407] Connection header received
0809 16:43:55.104118 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.104120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.104152 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.104220 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:55.104568 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46416: BlockingRecv error: recv got EOF from 127.0.0.1:46416 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 16:43:55.362741 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 16:43:55.362097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.28.62.189:45073)
0809 16:43:55.362175 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.362178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.362188 (+ 10us) server_negotiation.cc:407] Connection header received
0809 16:43:55.362220 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.362221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.362238 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.362310 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 16:43:55.362661 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36014: BlockingRecv error: recv got EOF from 127.0.0.1:36014 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 16:43:55.735050 23324 catalog_manager.cc:1259] Loaded cluster ID: b6d925d6f0e345c8b82e99b327928933
I20260809 16:43:55.735136 23324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: loading cluster ID for follower catalog manager: success
I20260809 16:43:55.735984 23324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: acquiring CA information for follower catalog manager: success
I20260809 16:43:55.736320 23324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0dad6191e8945a2a888b5eab7230f16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:43:55.784355 23362 catalog_manager.cc:1259] Loaded cluster ID: b6d925d6f0e345c8b82e99b327928933
I20260809 16:43:55.784448 23362 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: loading cluster ID for follower catalog manager: success
I20260809 16:43:55.785313 23362 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: acquiring CA information for follower catalog manager: success
I20260809 16:43:55.785650 23362 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c73f651a4190406c8e0b5596d885e524: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 16:43:55.878873 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:55.876114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.28.62.189:45073)
0809 16:43:55.876237 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.876239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.876253 (+ 14us) server_negotiation.cc:407] Connection header received
0809 16:43:55.876345 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.876346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.876400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.876478 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:55.876645 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.877141 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.877967 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.878188 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.878270 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:55.878453 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:55.878566 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:55.878629 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:55.878721 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 16:43:55.878750 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260809 16:43:55.883324 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:55.880798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.28.62.188:36013)
0809 16:43:55.880901 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.880903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.880928 (+ 25us) server_negotiation.cc:407] Connection header received
0809 16:43:55.880998 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.880999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.881036 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.881109 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:55.881282 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.881825 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.882560 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.882762 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.882817 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:55.882979 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:55.883151 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:55.883214 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:55.883239 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 16:43:55.883258 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20260809 16:43:55.888166 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:55.885154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.28.62.189:45073)
0809 16:43:55.885236 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.885238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.885363 (+ 125us) server_negotiation.cc:407] Connection header received
0809 16:43:55.885447 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.885448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.885458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.885509 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:55.885604 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.886298 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887389 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.887602 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887685 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:55.887866 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:55.887905 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:55.887960 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:55.888036 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 16:43:55.888061 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 16:43:55.888504 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:55.885174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.28.62.188:36013)
0809 16:43:55.885223 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.885225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.885430 (+ 205us) server_negotiation.cc:407] Connection header received
0809 16:43:55.885453 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.885455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.885464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.885509 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:55.885657 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.886179 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887371 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.887653 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887711 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:55.887994 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:55.888021 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:55.888064 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:55.888389 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 16:43:55.888415 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260809 16:43:55.888581 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 16:43:55.885043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.28.62.190:37075)
0809 16:43:55.885251 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 16:43:55.885252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0809 16:43:55.885474 (+ 222us) server_negotiation.cc:407] Connection header received
0809 16:43:55.885525 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 16:43:55.885526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 16:43:55.885591 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 16:43:55.885668 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 16:43:55.885855 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.886558 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887402 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 16:43:55.887725 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 16:43:55.887790 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 16:43:55.887975 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 16:43:55.888090 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 16:43:55.888147 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 16:43:55.888251 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 16:43:55.888493 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":53,"threads_started":1}
I20260809 16:43:55.890247 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 23022
I20260809 16:43:55.897883 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 23109
I20260809 16:43:55.904901 28922 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb9mnon/build/release/bin/kudu with pid 23195
2026-08-09T16:43:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2927 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5917 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280119 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST